Package org.apache.spark.resource

Class ExecutorResourceRequests

Object
org.apache.spark.resource.ExecutorResourceRequests
All Implemented Interfaces:
Serializable
public class ExecutorResourceRequests extends Object implements Serializable
A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
See Also:

  • Constructor Details

    • ExecutorResourceRequests

      public ExecutorResourceRequests()

  • Method Details

    • cores

      public ExecutorResourceRequests cores(int amount)
      Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.

      Parameters:
      amount - Number of cores to allocate per Executor.
      Returns:
      (undocumented)

    • memory

      public ExecutorResourceRequests memory(String amount)
      Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.

      Parameters:
      amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
      Returns:
      (undocumented)

    • memoryOverhead

      public ExecutorResourceRequests memoryOverhead(String amount)
      Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.

      Parameters:
      amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
      Returns:
      (undocumented)

    • offHeapMemory

      public ExecutorResourceRequests offHeapMemory(String amount)
      Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.

      Parameters:
      amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
      Returns:
      (undocumented)

    • pysparkMemory

      public ExecutorResourceRequests pysparkMemory(String amount)
      Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.

      Parameters:
      amount - Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
      Returns:
      (undocumented)

    • requests

      public scala.collection.immutable.Map<String,ExecutorResourceRequest> requests()
      Returns all the resource requests for the executor.
      Returns:
      (undocumented)

    • requestsJMap

      public Map<String,ExecutorResourceRequest> requestsJMap()
      (Java-specific) Returns all the resource requests for the executor.
      Returns:
      (undocumented)

    • resource

      public ExecutorResourceRequests resource(String resourceName, long amount, String discoveryScript, String vendor)
      Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.

      Parameters:
      resourceName - Name of the resource.
      amount - amount of that resource per executor to use.
      discoveryScript - Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.
      vendor - Optional vendor, required for some cluster managers
      Returns:
      (undocumented)

    • toString

      public String toString()
      Overrides:
      toString in class Object