Package org.apache.spark.resource

Class ResourceDiscoveryScriptPlugin

Object
org.apache.spark.resource.ResourceDiscoveryScriptPlugin
All Implemented Interfaces:
ResourceDiscoveryPlugin, org.apache.spark.internal.Logging
public class ResourceDiscoveryScriptPlugin extends Object implements ResourceDiscoveryPlugin, org.apache.spark.internal.Logging
The default plugin that is loaded into a Spark application to control how custom resources are discovered. This executes the discovery script specified by the user and gets the json output back and constructs ResourceInformation objects from that. If the user specifies custom plugins, this is the last one to be executed and throws if the resource isn't discovered.

Since:
3.0.0

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    ResourceDiscoveryScriptPlugin()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    Optional<ResourceInformation>
    discoverResource(ResourceRequest request, SparkConf sparkConf)
    Discover the addresses of the requested resource.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • ResourceDiscoveryScriptPlugin

      public ResourceDiscoveryScriptPlugin()

  • Method Details

    • discoverResource

      public Optional<ResourceInformation> discoverResource(ResourceRequest request, SparkConf sparkConf)
      Description copied from interface: ResourceDiscoveryPlugin
      Discover the addresses of the requested resource.

      This method is called early in the initialization of the Spark Executor/Driver/Worker. This function is responsible for discovering the addresses of the resource which Spark will then use for scheduling and eventually providing to the user. Depending on the deployment mode and and configuration of custom resources, this could be called by the Spark Driver, the Spark Executors, in standalone mode the Workers, or all of them. The ResourceRequest has a ResourceID component that can be used to distinguish which component it is called from and what resource its being called for. This will get called once for each resource type requested and its the responsibility of this function to return enough addresses of that resource based on the request. If the addresses do not meet the requested amount, Spark will fail. If this plugin doesn't handle a particular resource, it should return an empty Optional and Spark will try other plugins and then last fall back to the default discovery script plugin.

      Specified by:
      discoverResource in interface ResourceDiscoveryPlugin
      Parameters:
      request - The ResourceRequest that to be discovered.
      sparkConf - SparkConf
      Returns:
      An Optional containing a ResourceInformation object containing the resource name and the addresses of the resource. If it returns Optional.EMPTY other plugins will be called.