org.apache.spark.resource

Class ResourceDiscoveryScriptPlugin

Object

org.apache.spark.resource.ResourceDiscoveryScriptPlugin

All Implemented Interfaces:

ResourceDiscoveryPlugin, org.apache.spark.internal.Logging

public class ResourceDiscoveryScriptPlugin
extends Object
implements ResourceDiscoveryPlugin, org.apache.spark.internal.Logging

The default plugin that is loaded into a Spark application to control how custom resources are discovered. This executes the discovery script specified by the user and gets the json output back and constructs ResourceInformation objects from that. If the user specifies custom plugins, this is the last one to be executed and throws if the resource isn't discovered.

Since:

3.0.0

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
ResourceDiscoveryScriptPlugin()

Method Summary

Modifier and Type	Method and Description
java.util.Optional<ResourceInformation>	discoverResource(ResourceRequest request, SparkConf sparkConf) Discover the addresses of the requested resource.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

ResourceDiscoveryScriptPlugin

public ResourceDiscoveryScriptPlugin()

Method Detail

discoverResource

public java.util.Optional<ResourceInformation> discoverResource(ResourceRequest request,
                                                                SparkConf sparkConf)

Description copied from interface: ResourceDiscoveryPlugin

Discover the addresses of the requested resource.

This method is called early in the initialization of the Spark Executor/Driver/Worker. This function is responsible for discovering the addresses of the resource which Spark will then use for scheduling and eventually providing to the user. Depending on the deployment mode and and configuration of custom resources, this could be called by the Spark Driver, the Spark Executors, in standalone mode the Workers, or all of them. The ResourceRequest has a ResourceID component that can be used to distinguish which component it is called from and what resource its being called for. This will get called once for each resource type requested and its the responsibility of this function to return enough addresses of that resource based on the request. If the addresses do not meet the requested amount, Spark will fail. If this plugin doesn't handle a particular resource, it should return an empty Optional and Spark will try other plugins and then last fall back to the default discovery script plugin.

Specified by:

discoverResource in interface ResourceDiscoveryPlugin

Parameters:

request - The ResourceRequest that to be discovered.

sparkConf - SparkConf

Returns:

An Optional containing a ResourceInformation object containing the resource name and the addresses of the resource. If it returns Optional.EMPTY other plugins will be called.