Package org.apache.spark.api.plugin

Interface ExecutorPlugin

@DeveloperApi public interface ExecutorPlugin
:: DeveloperApi :: Executor component of a SparkPlugin.
Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default void
    init(PluginContext ctx, Map<String,String> extraConf)
    Initialize the executor plugin.
    default void
    onTaskFailed(TaskFailedReason failureReason)
    Perform an action after tasks completes with exceptions.
    default void
    onTaskStart()
    Perform any action before the task is run.
    default void
    onTaskSucceeded()
    Perform an action after tasks completes without exceptions.
    default void
    shutdown()
    Clean up and terminate this plugin.

  • Method Details

    • init

      default void init(PluginContext ctx, Map<String,String> extraConf)
      Initialize the executor plugin.

      When a Spark plugin provides an executor plugin, this method will be called during the initialization of the executor process. It will block executor initialization until it returns.

      Executor plugins that publish metrics should register all metrics with the context's registry (PluginContext.metricRegistry()) when this method is called. Metrics registered afterwards are not guaranteed to show up.

      Parameters:
      ctx - Context information for the executor where the plugin is running.
      extraConf - Extra configuration provided by the driver component during its initialization.

    • shutdown

      default void shutdown()
      Clean up and terminate this plugin.

      This method is called during the executor shutdown phase, and blocks executor shutdown.

    • onTaskStart

      default void onTaskStart()
      Perform any action before the task is run.

      This method is invoked from the same thread the task will be executed. Task-specific information can be accessed via TaskContext.get().

      Plugin authors should avoid expensive operations here, as this method will be called on every task, and doing something expensive can significantly slow down a job. It is not recommended for a user to call a remote service, for example.

      Exceptions thrown from this method do not propagate - they're caught, logged, and suppressed. Therefore exceptions when executing this method won't make the job fail.

      Since:
      3.1.0

    • onTaskSucceeded

      default void onTaskSucceeded()
      Perform an action after tasks completes without exceptions.

      As onTaskStart exceptions are suppressed, this method will still be invoked even if the corresponding onTaskStart() call for this task failed.

      Same warnings of onTaskStart apply here.

      Since:
      3.1.0

    • onTaskFailed

      default void onTaskFailed(TaskFailedReason failureReason)
      Perform an action after tasks completes with exceptions.

      Same warnings of onTaskStart apply here.

      Parameters:
      failureReason - the exception thrown from the failed task.
      Since:
      3.1.0