Package org.apache.spark.scheduler

Interface SparkListenerInterface

All Known Implementing Classes:
SparkFirehoseListener, SparkListener, SpillListener, StatsReportListener
public interface SparkListenerInterface
Interface for listening to events from the Spark scheduler. Most applications should probably extend SparkListener or SparkFirehoseListener directly, rather than implementing this class.

Note that this is an internal interface which might change in different Spark releases.

  • Method Details

    • onApplicationEnd

      void onApplicationEnd(SparkListenerApplicationEnd applicationEnd)
      Called when the application ends
      Parameters:
      applicationEnd - (undocumented)

    • onApplicationStart

      void onApplicationStart(SparkListenerApplicationStart applicationStart)
      Called when the application starts
      Parameters:
      applicationStart - (undocumented)

    • onBlockManagerAdded

      void onBlockManagerAdded(SparkListenerBlockManagerAdded blockManagerAdded)
      Called when a new block manager has joined
      Parameters:
      blockManagerAdded - (undocumented)

    • onBlockManagerRemoved

      void onBlockManagerRemoved(SparkListenerBlockManagerRemoved blockManagerRemoved)
      Called when an existing block manager has been removed
      Parameters:
      blockManagerRemoved - (undocumented)

    • onBlockUpdated

      void onBlockUpdated(SparkListenerBlockUpdated blockUpdated)
      Called when the driver receives a block update info.
      Parameters:
      blockUpdated - (undocumented)

    • onEnvironmentUpdate

      void onEnvironmentUpdate(SparkListenerEnvironmentUpdate environmentUpdate)
      Called when environment properties have been updated
      Parameters:
      environmentUpdate - (undocumented)

    • onExecutorAdded

      void onExecutorAdded(SparkListenerExecutorAdded executorAdded)
      Called when the driver registers a new executor.
      Parameters:
      executorAdded - (undocumented)

    • onExecutorBlacklisted

      void onExecutorBlacklisted(SparkListenerExecutorBlacklisted executorBlacklisted)
      Deprecated.
      use onExecutorExcluded instead. Since 3.1.0.
      Called when the driver excludes an executor for a Spark application.
      Parameters:
      executorBlacklisted - (undocumented)

    • onExecutorBlacklistedForStage

      void onExecutorBlacklistedForStage(SparkListenerExecutorBlacklistedForStage executorBlacklistedForStage)
      Deprecated.
      use onExecutorExcludedForStage instead. Since 3.1.0.
      Called when the driver excludes an executor for a stage.
      Parameters:
      executorBlacklistedForStage - (undocumented)

    • onExecutorExcluded

      void onExecutorExcluded(SparkListenerExecutorExcluded executorExcluded)
      Called when the driver excludes an executor for a Spark application.
      Parameters:
      executorExcluded - (undocumented)

    • onExecutorExcludedForStage

      void onExecutorExcludedForStage(SparkListenerExecutorExcludedForStage executorExcludedForStage)
      Called when the driver excludes an executor for a stage.
      Parameters:
      executorExcludedForStage - (undocumented)

    • onExecutorMetricsUpdate

      void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate executorMetricsUpdate)
      Called when the driver receives task metrics from an executor in a heartbeat.
      Parameters:
      executorMetricsUpdate - (undocumented)

    • onExecutorRemoved

      void onExecutorRemoved(SparkListenerExecutorRemoved executorRemoved)
      Called when the driver removes an executor.
      Parameters:
      executorRemoved - (undocumented)

    • onExecutorUnblacklisted

      void onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted executorUnblacklisted)
      Deprecated.
      use onExecutorUnexcluded instead. Since 3.1.0.
      Called when the driver re-enables a previously excluded executor.
      Parameters:
      executorUnblacklisted - (undocumented)

    • onExecutorUnexcluded

      void onExecutorUnexcluded(SparkListenerExecutorUnexcluded executorUnexcluded)
      Called when the driver re-enables a previously excluded executor.
      Parameters:
      executorUnexcluded - (undocumented)

    • onJobEnd

      void onJobEnd(SparkListenerJobEnd jobEnd)
      Called when a job ends
      Parameters:
      jobEnd - (undocumented)

    • onJobStart

      void onJobStart(SparkListenerJobStart jobStart)
      Called when a job starts
      Parameters:
      jobStart - (undocumented)

    • onNodeBlacklisted

      void onNodeBlacklisted(SparkListenerNodeBlacklisted nodeBlacklisted)
      Deprecated.
      use onNodeExcluded instead. Since 3.1.0.
      Called when the driver excludes a node for a Spark application.
      Parameters:
      nodeBlacklisted - (undocumented)

    • onNodeBlacklistedForStage

      void onNodeBlacklistedForStage(SparkListenerNodeBlacklistedForStage nodeBlacklistedForStage)
      Deprecated.
      use onNodeExcludedForStage instead. Since 3.1.0.
      Called when the driver excludes a node for a stage.
      Parameters:
      nodeBlacklistedForStage - (undocumented)

    • onNodeExcluded

      void onNodeExcluded(SparkListenerNodeExcluded nodeExcluded)
      Called when the driver excludes a node for a Spark application.
      Parameters:
      nodeExcluded - (undocumented)

    • onNodeExcludedForStage

      void onNodeExcludedForStage(SparkListenerNodeExcludedForStage nodeExcludedForStage)
      Called when the driver excludes a node for a stage.
      Parameters:
      nodeExcludedForStage - (undocumented)

    • onNodeUnblacklisted

      void onNodeUnblacklisted(SparkListenerNodeUnblacklisted nodeUnblacklisted)
      Deprecated.
      use onNodeUnexcluded instead. Since 3.1.0.
      Called when the driver re-enables a previously excluded node.
      Parameters:
      nodeUnblacklisted - (undocumented)

    • onNodeUnexcluded

      void onNodeUnexcluded(SparkListenerNodeUnexcluded nodeUnexcluded)
      Called when the driver re-enables a previously excluded node.
      Parameters:
      nodeUnexcluded - (undocumented)

    • onOtherEvent

      void onOtherEvent(SparkListenerEvent event)
      Called when other events like SQL-specific events are posted.
      Parameters:
      event - (undocumented)

    • onResourceProfileAdded

      void onResourceProfileAdded(SparkListenerResourceProfileAdded event)
      Called when a Resource Profile is added to the manager.
      Parameters:
      event - (undocumented)

    • onSpeculativeTaskSubmitted

      void onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted speculativeTask)
      Called when a speculative task is submitted
      Parameters:
      speculativeTask - (undocumented)

    • onStageCompleted

      void onStageCompleted(SparkListenerStageCompleted stageCompleted)
      Called when a stage completes successfully or fails, with information on the completed stage.
      Parameters:
      stageCompleted - (undocumented)

    • onStageExecutorMetrics

      void onStageExecutorMetrics(SparkListenerStageExecutorMetrics executorMetrics)
      Called with the peak memory metrics for a given (executor, stage) combination. Note that this is only present when reading from the event log (as in the history server), and is never called in a live application.
      Parameters:
      executorMetrics - (undocumented)

    • onStageSubmitted

      void onStageSubmitted(SparkListenerStageSubmitted stageSubmitted)
      Called when a stage is submitted
      Parameters:
      stageSubmitted - (undocumented)

    • onTaskEnd

      void onTaskEnd(SparkListenerTaskEnd taskEnd)
      Called when a task ends
      Parameters:
      taskEnd - (undocumented)

    • onTaskGettingResult

      void onTaskGettingResult(SparkListenerTaskGettingResult taskGettingResult)
      Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
      Parameters:
      taskGettingResult - (undocumented)

    • onTaskStart

      void onTaskStart(SparkListenerTaskStart taskStart)
      Called when a task starts
      Parameters:
      taskStart - (undocumented)

    • onUnpersistRDD

      void onUnpersistRDD(SparkListenerUnpersistRDD unpersistRDD)
      Called when an RDD is manually unpersisted by the application
      Parameters:
      unpersistRDD - (undocumented)

    • onUnschedulableTaskSetAdded

      void onUnschedulableTaskSetAdded(SparkListenerUnschedulableTaskSetAdded unschedulableTaskSetAdded)
      Called when a taskset becomes unschedulable due to exludeOnFailure and dynamic allocation is enabled.
      Parameters:
      unschedulableTaskSetAdded - (undocumented)

    • onUnschedulableTaskSetRemoved

      void onUnschedulableTaskSetRemoved(SparkListenerUnschedulableTaskSetRemoved unschedulableTaskSetRemoved)
      Called when an unschedulable taskset becomes schedulable and dynamic allocation is enabled.
      Parameters:
      unschedulableTaskSetRemoved - (undocumented)