class ExecutorResourceRequests extends Serializable
A set of Executor resource requests. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
- Annotations
- @Evolving() @Since("3.1.0")
- Source
- ExecutorResourceRequests.scala
- Alphabetic
- By Inheritance
- ExecutorResourceRequests
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ExecutorResourceRequests()
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def cores(amount: Int): ExecutorResourceRequests.this.type
Specify number of cores per Executor.
Specify number of cores per Executor. This is a convenient API to add ExecutorResourceRequest for "cores" resource.
- amount
Number of cores to allocate per Executor.
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def memory(amount: String): ExecutorResourceRequests.this.type
Specify heap memory.
Specify heap memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memory" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
- def memoryOverhead(amount: String): ExecutorResourceRequests.this.type
Specify overhead memory.
Specify overhead memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "memoryOverhead" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def offHeapMemory(amount: String): ExecutorResourceRequests.this.type
Specify off heap memory.
Specify off heap memory. The value specified will be converted to MiB. This value only take effect when MEMORY_OFFHEAP_ENABLED is true. This is a convenient API to add ExecutorResourceRequest for "offHeap" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
- def pysparkMemory(amount: String): ExecutorResourceRequests.this.type
Specify pyspark memory.
Specify pyspark memory. The value specified will be converted to MiB. This is a convenient API to add ExecutorResourceRequest for "pyspark.memory" resource.
- amount
Amount of memory. In the same format as JVM memory strings (e.g. 512m, 2g). Default unit is MiB if not specified.
- def requests: Map[String, ExecutorResourceRequest]
Returns all the resource requests for the executor.
- def requestsJMap: Map[String, ExecutorResourceRequest]
(Java-specific) Returns all the resource requests for the executor.
- def resource(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = ""): ExecutorResourceRequests.this.type
Amount of a particular custom resource(GPU, FPGA, etc) to use.
Amount of a particular custom resource(GPU, FPGA, etc) to use. The resource names supported correspond to the regular Spark configs with the prefix removed. For instance, resources like GPUs are gpu (spark configs spark.executor.resource.gpu.*). If you pass in a resource that the cluster manager doesn't support the result is undefined, it may error or may just be ignored. This is a convenient API to add ExecutorResourceRequest for custom resources.
- resourceName
Name of the resource.
- amount
amount of that resource per executor to use.
- discoveryScript
Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to of the resources available.
- vendor
Optional vendor, required for some cluster managers
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- ExecutorResourceRequests → AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)