Package org.apache.spark.scheduler

Interface SchedulerBackend

public interface SchedulerBackend
A backend interface for scheduling systems that allows plugging in different ones under TaskSchedulerImpl. We assume a model where the application gets resource offers as machines become available and can launch tasks on them.

  • Method Details

    • applicationAttemptId

      scala.Option<String> applicationAttemptId()
      Get the attempt ID for this run, if the cluster manager supports multiple attempts. Applications run in client mode will not have attempt IDs.

      Returns:
      The application attempt id, if available.

    • applicationId

      String applicationId()
      Get an application ID associated with the job.

      Returns:
      An application ID

    • defaultParallelism

      int defaultParallelism()

    • getDriverAttributes

      scala.Option<scala.collection.immutable.Map<String,String>> getDriverAttributes()
      Get the attributes on driver. These attributes are used to replace log URLs when custom log url pattern is specified.
      Returns:
      Map containing attributes on driver.

    • getDriverLogUrls

      scala.Option<scala.collection.immutable.Map<String,String>> getDriverLogUrls()
      Get the URLs for the driver logs. These URLs are used to display the links in the UI Executors tab for the driver.
      Returns:
      Map containing the log names and their respective URLs

    • getShufflePushMergerLocations

      scala.collection.immutable.Seq<BlockManagerId> getShufflePushMergerLocations(int numPartitions, int resourceProfileId)
      Get the list of host locations for push based shuffle

      Currently push based shuffle is disabled for both stage retry and stage reuse cases (for eg: in the case where few partitions are lost due to failure). Hence this method should be invoked only once for a ShuffleDependency.

      Parameters:
      numPartitions - (undocumented)
      resourceProfileId - (undocumented)
      Returns:
      List of external shuffle services locations

    • getTaskThreadDump

      scala.Option<ThreadStackTrace> getTaskThreadDump(long taskId, String executorId)

    • isReady

      boolean isReady()

    • killTask

      void killTask(long taskId, String executorId, boolean interruptThread, String reason)
      Requests that an executor kills a running task.

      Parameters:
      taskId - Id of the task.
      executorId - Id of the executor the task is running on.
      interruptThread - Whether the executor should interrupt the task thread.
      reason - The reason for the task kill.

    • maxNumConcurrentTasks

      int maxNumConcurrentTasks(ResourceProfile rp)
      Get the max number of tasks that can be concurrent launched based on the ResourceProfile could be used, even if some of them are being used at the moment. Note that please don't cache the value returned by this method, because the number can change due to add/remove executors.

      Parameters:
      rp - ResourceProfile which to use to calculate max concurrent tasks.
      Returns:
      The max number of tasks that can be concurrent launched currently.

    • reviveOffers

      void reviveOffers()
      Update the current offers and schedule tasks

    • start

      void start()

    • stop

      void stop()

    • stop

      void stop(int exitCode)

    • updateExecutorsLogLevel

      void updateExecutorsLogLevel(String logLevel)