Packages

c

org.apache.spark.resource

ExecutorResourceRequests

class ExecutorResourceRequests extends Serializable

A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.

Annotations
@Evolving() @Since( "3.1.0" )
Source
ExecutorResourceRequests.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExecutorResourceRequests
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExecutorResourceRequests()

Value Members

  1. def cores(amount: Int): ExecutorResourceRequests.this.type

    Specify number of cores per Executor.

    Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.

    amount

    Number of cores to allocate per Executor.

  2. def memory(amount: String): ExecutorResourceRequests.this.type

    Specify heap memory.

    Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  3. def memoryOverhead(amount: String): ExecutorResourceRequests.this.type

    Specify overhead memory.

    Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  4. def offHeapMemory(amount: String): ExecutorResourceRequests.this.type

    Specify off heap memory.

    Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  5. def pysparkMemory(amount: String): ExecutorResourceRequests.this.type

    Specify pyspark memory.

    Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  6. def requests: Map[String, ExecutorResourceRequest]

    Returns all the resource requests for the executor.

  7. def requestsJMap: Map[String, ExecutorResourceRequest]

    (Java-specific) Returns all the resource requests for the executor.

  8. def resource(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = ""): ExecutorResourceRequests.this.type

    Amount of a particular custom resource(GPU, FPGA, etc) to use.

    Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.

    resourceName

    Name of the resource.

    amount

    amount of that resource per executor to use.

    discoveryScript

    Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.

    vendor

    Optional vendor, required for some cluster managers

  9. def toString(): String
    Definition Classes
    ExecutorResourceRequests → AnyRef → Any