Package org.apache.spark
Class SparkFirehoseListener
Object
org.apache.spark.SparkFirehoseListener
- All Implemented Interfaces:
SparkListenerInterface
Class that allows users to receive all SparkListener events.
Users should override the onEvent method.
This is a concrete Java class in order to ensure that we don't forget to update it when adding
new methods to SparkListener: forgetting to add a method will result in a compilation error (if
this was a concrete Scala class, default implementations of new event handlers would be inherited
from the SparkListener trait).
Please note until Spark 3.1.0 this was missing the DevelopApi annotation, this needs to be
taken into account if changing this API before a major release.
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionfinal void
onApplicationEnd
(SparkListenerApplicationEnd applicationEnd) Called when the application endsfinal void
onApplicationStart
(SparkListenerApplicationStart applicationStart) Called when the application startsfinal void
onBlockManagerAdded
(SparkListenerBlockManagerAdded blockManagerAdded) Called when a new block manager has joinedfinal void
onBlockManagerRemoved
(SparkListenerBlockManagerRemoved blockManagerRemoved) Called when an existing block manager has been removedvoid
onBlockUpdated
(SparkListenerBlockUpdated blockUpdated) Called when the driver receives a block update info.final void
onEnvironmentUpdate
(SparkListenerEnvironmentUpdate environmentUpdate) Called when environment properties have been updatedvoid
onEvent
(SparkListenerEvent event) final void
onExecutorAdded
(SparkListenerExecutorAdded executorAdded) Called when the driver registers a new executor.final void
onExecutorBlacklisted
(SparkListenerExecutorBlacklisted executorBlacklisted) Called when the driver excludes an executor for a Spark application.void
onExecutorBlacklistedForStage
(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage) Called when the driver excludes an executor for a stage.final void
onExecutorExcluded
(SparkListenerExecutorExcluded executorExcluded) Called when the driver excludes an executor for a Spark application.void
onExecutorExcludedForStage
(SparkListenerExecutorExcludedForStage executorExcludedForStage) Called when the driver excludes an executor for a stage.final void
onExecutorMetricsUpdate
(SparkListenerExecutorMetricsUpdate executorMetricsUpdate) Called when the driver receives task metrics from an executor in a heartbeat.final void
onExecutorRemoved
(SparkListenerExecutorRemoved executorRemoved) Called when the driver removes an executor.final void
onExecutorUnblacklisted
(SparkListenerExecutorUnblacklisted executorUnblacklisted) Called when the driver re-enables a previously excluded executor.final void
onExecutorUnexcluded
(SparkListenerExecutorUnexcluded executorUnexcluded) Called when the driver re-enables a previously excluded executor.final void
onJobEnd
(SparkListenerJobEnd jobEnd) Called when a job endsfinal void
onJobStart
(SparkListenerJobStart jobStart) Called when a job startsfinal void
onNodeBlacklisted
(SparkListenerNodeBlacklisted nodeBlacklisted) Called when the driver excludes a node for a Spark application.void
onNodeBlacklistedForStage
(SparkListenerNodeBlacklistedForStage nodeBlacklistedForStage) Called when the driver excludes a node for a stage.final void
onNodeExcluded
(SparkListenerNodeExcluded nodeExcluded) Called when the driver excludes a node for a Spark application.void
onNodeExcludedForStage
(SparkListenerNodeExcludedForStage nodeExcludedForStage) Called when the driver excludes a node for a stage.final void
onNodeUnblacklisted
(SparkListenerNodeUnblacklisted nodeUnblacklisted) Called when the driver re-enables a previously excluded node.final void
onNodeUnexcluded
(SparkListenerNodeUnexcluded nodeUnexcluded) Called when the driver re-enables a previously excluded node.void
onOtherEvent
(SparkListenerEvent event) Called when other events like SQL-specific events are posted.void
Called when a Resource Profile is added to the manager.void
onSpeculativeTaskSubmitted
(SparkListenerSpeculativeTaskSubmitted speculativeTask) Called when a speculative task is submittedfinal void
onStageCompleted
(SparkListenerStageCompleted stageCompleted) Called when a stage completes successfully or fails, with information on the completed stage.final void
onStageExecutorMetrics
(SparkListenerStageExecutorMetrics executorMetrics) Called with the peak memory metrics for a given (executor, stage) combination.final void
onStageSubmitted
(SparkListenerStageSubmitted stageSubmitted) Called when a stage is submittedfinal void
onTaskEnd
(SparkListenerTaskEnd taskEnd) Called when a task endsfinal void
onTaskGettingResult
(SparkListenerTaskGettingResult taskGettingResult) Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).final void
onTaskStart
(SparkListenerTaskStart taskStart) Called when a task startsfinal void
onUnpersistRDD
(SparkListenerUnpersistRDD unpersistRDD) Called when an RDD is manually unpersisted by the applicationvoid
onUnschedulableTaskSetAdded
(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded) Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.void
onUnschedulableTaskSetRemoved
(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved) Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.
-
Constructor Details
-
SparkFirehoseListener
public SparkFirehoseListener()
-
-
Method Details
-
onEvent
-
onStageCompleted
Description copied from interface:SparkListenerInterface
Called when a stage completes successfully or fails, with information on the completed stage.- Specified by:
onStageCompleted
in interfaceSparkListenerInterface
- Parameters:
stageCompleted
- (undocumented)
-
onStageSubmitted
Description copied from interface:SparkListenerInterface
Called when a stage is submitted- Specified by:
onStageSubmitted
in interfaceSparkListenerInterface
- Parameters:
stageSubmitted
- (undocumented)
-
onTaskStart
Description copied from interface:SparkListenerInterface
Called when a task starts- Specified by:
onTaskStart
in interfaceSparkListenerInterface
- Parameters:
taskStart
- (undocumented)
-
onTaskGettingResult
Description copied from interface:SparkListenerInterface
Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).- Specified by:
onTaskGettingResult
in interfaceSparkListenerInterface
- Parameters:
taskGettingResult
- (undocumented)
-
onTaskEnd
Description copied from interface:SparkListenerInterface
Called when a task ends- Specified by:
onTaskEnd
in interfaceSparkListenerInterface
- Parameters:
taskEnd
- (undocumented)
-
onJobStart
Description copied from interface:SparkListenerInterface
Called when a job starts- Specified by:
onJobStart
in interfaceSparkListenerInterface
- Parameters:
jobStart
- (undocumented)
-
onJobEnd
Description copied from interface:SparkListenerInterface
Called when a job ends- Specified by:
onJobEnd
in interfaceSparkListenerInterface
- Parameters:
jobEnd
- (undocumented)
-
onEnvironmentUpdate
Description copied from interface:SparkListenerInterface
Called when environment properties have been updated- Specified by:
onEnvironmentUpdate
in interfaceSparkListenerInterface
- Parameters:
environmentUpdate
- (undocumented)
-
onBlockManagerAdded
Description copied from interface:SparkListenerInterface
Called when a new block manager has joined- Specified by:
onBlockManagerAdded
in interfaceSparkListenerInterface
- Parameters:
blockManagerAdded
- (undocumented)
-
onBlockManagerRemoved
Description copied from interface:SparkListenerInterface
Called when an existing block manager has been removed- Specified by:
onBlockManagerRemoved
in interfaceSparkListenerInterface
- Parameters:
blockManagerRemoved
- (undocumented)
-
onUnpersistRDD
Description copied from interface:SparkListenerInterface
Called when an RDD is manually unpersisted by the application- Specified by:
onUnpersistRDD
in interfaceSparkListenerInterface
- Parameters:
unpersistRDD
- (undocumented)
-
onApplicationStart
Description copied from interface:SparkListenerInterface
Called when the application starts- Specified by:
onApplicationStart
in interfaceSparkListenerInterface
- Parameters:
applicationStart
- (undocumented)
-
onApplicationEnd
Description copied from interface:SparkListenerInterface
Called when the application ends- Specified by:
onApplicationEnd
in interfaceSparkListenerInterface
- Parameters:
applicationEnd
- (undocumented)
-
onExecutorMetricsUpdate
Description copied from interface:SparkListenerInterface
Called when the driver receives task metrics from an executor in a heartbeat.- Specified by:
onExecutorMetricsUpdate
in interfaceSparkListenerInterface
- Parameters:
executorMetricsUpdate
- (undocumented)
-
onStageExecutorMetrics
Description copied from interface:SparkListenerInterface
Called with the peak memory metrics for a given (executor, stage) combination. Note that this is only present when reading from the event log (as in the history server), and is never called in a live application.- Specified by:
onStageExecutorMetrics
in interfaceSparkListenerInterface
- Parameters:
executorMetrics
- (undocumented)
-
onExecutorAdded
Description copied from interface:SparkListenerInterface
Called when the driver registers a new executor.- Specified by:
onExecutorAdded
in interfaceSparkListenerInterface
- Parameters:
executorAdded
- (undocumented)
-
onExecutorRemoved
Description copied from interface:SparkListenerInterface
Called when the driver removes an executor.- Specified by:
onExecutorRemoved
in interfaceSparkListenerInterface
- Parameters:
executorRemoved
- (undocumented)
-
onExecutorBlacklisted
Description copied from interface:SparkListenerInterface
Called when the driver excludes an executor for a Spark application.- Specified by:
onExecutorBlacklisted
in interfaceSparkListenerInterface
- Parameters:
executorBlacklisted
- (undocumented)
-
onExecutorExcluded
Description copied from interface:SparkListenerInterface
Called when the driver excludes an executor for a Spark application.- Specified by:
onExecutorExcluded
in interfaceSparkListenerInterface
- Parameters:
executorExcluded
- (undocumented)
-
onExecutorBlacklistedForStage
public void onExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage) Description copied from interface:SparkListenerInterface
Called when the driver excludes an executor for a stage.- Specified by:
onExecutorBlacklistedForStage
in interfaceSparkListenerInterface
- Parameters:
executorBlacklistedForStage
- (undocumented)
-
onExecutorExcludedForStage
public void onExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage) Description copied from interface:SparkListenerInterface
Called when the driver excludes an executor for a stage.- Specified by:
onExecutorExcludedForStage
in interfaceSparkListenerInterface
- Parameters:
executorExcludedForStage
- (undocumented)
-
onNodeBlacklistedForStage
Description copied from interface:SparkListenerInterface
Called when the driver excludes a node for a stage.- Specified by:
onNodeBlacklistedForStage
in interfaceSparkListenerInterface
- Parameters:
nodeBlacklistedForStage
- (undocumented)
-
onNodeExcludedForStage
Description copied from interface:SparkListenerInterface
Called when the driver excludes a node for a stage.- Specified by:
onNodeExcludedForStage
in interfaceSparkListenerInterface
- Parameters:
nodeExcludedForStage
- (undocumented)
-
onExecutorUnblacklisted
Description copied from interface:SparkListenerInterface
Called when the driver re-enables a previously excluded executor.- Specified by:
onExecutorUnblacklisted
in interfaceSparkListenerInterface
- Parameters:
executorUnblacklisted
- (undocumented)
-
onExecutorUnexcluded
Description copied from interface:SparkListenerInterface
Called when the driver re-enables a previously excluded executor.- Specified by:
onExecutorUnexcluded
in interfaceSparkListenerInterface
- Parameters:
executorUnexcluded
- (undocumented)
-
onNodeBlacklisted
Description copied from interface:SparkListenerInterface
Called when the driver excludes a node for a Spark application.- Specified by:
onNodeBlacklisted
in interfaceSparkListenerInterface
- Parameters:
nodeBlacklisted
- (undocumented)
-
onNodeExcluded
Description copied from interface:SparkListenerInterface
Called when the driver excludes a node for a Spark application.- Specified by:
onNodeExcluded
in interfaceSparkListenerInterface
- Parameters:
nodeExcluded
- (undocumented)
-
onNodeUnblacklisted
Description copied from interface:SparkListenerInterface
Called when the driver re-enables a previously excluded node.- Specified by:
onNodeUnblacklisted
in interfaceSparkListenerInterface
- Parameters:
nodeUnblacklisted
- (undocumented)
-
onNodeUnexcluded
Description copied from interface:SparkListenerInterface
Called when the driver re-enables a previously excluded node.- Specified by:
onNodeUnexcluded
in interfaceSparkListenerInterface
- Parameters:
nodeUnexcluded
- (undocumented)
-
onBlockUpdated
Description copied from interface:SparkListenerInterface
Called when the driver receives a block update info.- Specified by:
onBlockUpdated
in interfaceSparkListenerInterface
- Parameters:
blockUpdated
- (undocumented)
-
onSpeculativeTaskSubmitted
Description copied from interface:SparkListenerInterface
Called when a speculative task is submitted- Specified by:
onSpeculativeTaskSubmitted
in interfaceSparkListenerInterface
- Parameters:
speculativeTask
- (undocumented)
-
onUnschedulableTaskSetAdded
public void onUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded) Description copied from interface:SparkListenerInterface
Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.- Specified by:
onUnschedulableTaskSetAdded
in interfaceSparkListenerInterface
- Parameters:
unschedulableTaskSetAdded
- (undocumented)
-
onUnschedulableTaskSetRemoved
public void onUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved) Description copied from interface:SparkListenerInterface
Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.- Specified by:
onUnschedulableTaskSetRemoved
in interfaceSparkListenerInterface
- Parameters:
unschedulableTaskSetRemoved
- (undocumented)
-
onResourceProfileAdded
Description copied from interface:SparkListenerInterface
Called when a Resource Profile is added to the manager.- Specified by:
onResourceProfileAdded
in interfaceSparkListenerInterface
- Parameters:
event
- (undocumented)
-
onOtherEvent
Description copied from interface:SparkListenerInterface
Called when other events like SQL-specific events are posted.- Specified by:
onOtherEvent
in interfaceSparkListenerInterface
- Parameters:
event
- (undocumented)
-