org.apache.spark.scheduler

Interface ExternalClusterManager

public interface ExternalClusterManager

A cluster manager interface to plugin external scheduler.

Method Summary

Modifier and Type	Method and Description
boolean	canCreate(String masterURL) Check if this cluster manager instance can create scheduler components for a certain master URL.
SchedulerBackend	createSchedulerBackend(SparkContext sc, String masterURL, TaskScheduler scheduler) Create a scheduler backend for the given SparkContext and scheduler.
TaskScheduler	createTaskScheduler(SparkContext sc, String masterURL) Create a task scheduler instance for the given SparkContext
void	initialize(TaskScheduler scheduler, SchedulerBackend backend) Initialize task scheduler and backend scheduler.

Method Detail

canCreate

boolean canCreate(String masterURL)

Check if this cluster manager instance can create scheduler components for a certain master URL.

Parameters:

masterURL - the master URL

Returns:

True if the cluster manager can create scheduler backend/

createSchedulerBackend

SchedulerBackend createSchedulerBackend(SparkContext sc,
                                        String masterURL,
                                        TaskScheduler scheduler)

Create a scheduler backend for the given SparkContext and scheduler. This is called after task scheduler is created using ExternalClusterManager.createTaskScheduler().

Parameters:

sc - SparkContext

masterURL - the master URL

scheduler - TaskScheduler that will be used with the scheduler backend.

Returns:

SchedulerBackend that works with a TaskScheduler

createTaskScheduler

TaskScheduler createTaskScheduler(SparkContext sc,
                                  String masterURL)

Create a task scheduler instance for the given SparkContext

Parameters:

sc - SparkContext

masterURL - the master URL

Returns:

TaskScheduler that will be responsible for task handling

initialize

void initialize(TaskScheduler scheduler,
                SchedulerBackend backend)

Initialize task scheduler and backend scheduler. This is called after the scheduler components are created

Parameters:

scheduler - TaskScheduler that will be responsible for task handling

backend - SchedulerBackend that works with a TaskScheduler