Packages

c

org.apache.spark.resource

ExecutorResourceRequests

class ExecutorResourceRequests extends Serializable

A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.

Annotations
@Evolving() @Since( "3.1.0" )
Source
ExecutorResourceRequests.scala
Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ExecutorResourceRequests
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new ExecutorResourceRequests()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  6. def cores(amount: Int): ExecutorResourceRequests.this.type

    Specify number of cores per Executor.

    Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.

    amount

    Number of cores to allocate per Executor.

  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  13. def memory(amount: String): ExecutorResourceRequests.this.type

    Specify heap memory.

    Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  14. def memoryOverhead(amount: String): ExecutorResourceRequests.this.type

    Specify overhead memory.

    Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  15. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  16. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  18. def offHeapMemory(amount: String): ExecutorResourceRequests.this.type

    Specify off heap memory.

    Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  19. def pysparkMemory(amount: String): ExecutorResourceRequests.this.type

    Specify pyspark memory.

    Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.

    amount

    Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.

  20. def requests: Map[String, ExecutorResourceRequest]

    Returns all the resource requests for the executor.

  21. def requestsJMap: Map[String, ExecutorResourceRequest]

    (Java-specific) Returns all the resource requests for the executor.

  22. def resource(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = ""): ExecutorResourceRequests.this.type

    Amount of a particular custom resource(GPU, FPGA, etc) to use.

    Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.

    resourceName

    Name of the resource.

    amount

    amount of that resource per executor to use.

    discoveryScript

    Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.

    vendor

    Optional vendor, required for some cluster managers

  23. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  24. def toString(): String
    Definition Classes
    ExecutorResourceRequests → AnyRef → Any
  25. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  26. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped