Package org.apache.spark.api.resource

Interface ResourceDiscoveryPlugin

All Known Implementing Classes:
ResourceDiscoveryScriptPlugin
@DeveloperApi public interface ResourceDiscoveryPlugin
:: DeveloperApi :: A plugin that can be dynamically loaded into a Spark application to control how custom resources are discovered. Plugins can be chained to allow different plugins to handle different resource types.

Plugins must implement the function discoveryResource.

Since:
3.0.0

  • Method Details

    • discoverResource

      Optional<ResourceInformation> discoverResource(ResourceRequest request, SparkConf sparkConf)
      Discover the addresses of the requested resource.

      This method is called early in the initialization of the Spark Executor/Driver/Worker. This function is responsible for discovering the addresses of the resource which Spark will then use for scheduling and eventually providing to the user. Depending on the deployment mode and and configuration of custom resources, this could be called by the Spark Driver, the Spark Executors, in standalone mode the Workers, or all of them. The ResourceRequest has a ResourceID component that can be used to distinguish which component it is called from and what resource its being called for. This will get called once for each resource type requested and its the responsibility of this function to return enough addresses of that resource based on the request. If the addresses do not meet the requested amount, Spark will fail. If this plugin doesn't handle a particular resource, it should return an empty Optional and Spark will try other plugins and then last fall back to the default discovery script plugin.

      Parameters:
      request - The ResourceRequest that to be discovered.
      sparkConf - SparkConf
      Returns:
      An Optional containing a ResourceInformation object containing the resource name and the addresses of the resource. If it returns Optional.EMPTY other plugins will be called.