class ExecutorResourceRequest extends Serializable
An Executor resource request. This is used in conjunction with the ResourceProfile to programmatically specify the resources needed for an RDD that will be applied at the stage level.
This is used to specify what the resource requirements are for an Executor and how Spark can find out specific details about those resources. Not all the parameters are required for every resource type. Resources like GPUs are supported and have same limitations as using the global spark configs spark.executor.resource.gpu.*. The amount, discoveryScript, and vendor parameters for resources are all the same parameters a user would specify through the configs: spark.executor.resource.{resourceName}.{amount, discoveryScript, vendor}.
For instance, a user wants to allocate an Executor with GPU resources on YARN. The user has to specify the resource name (gpu), the amount or number of GPUs per Executor, the discovery script would be specified so that when the Executor starts up it can discovery what GPU addresses are available for it to use because YARN doesn't tell Spark that, then vendor would not be used because its specific for Kubernetes.
See the configuration and cluster specific docs for more details.
Use ExecutorResourceRequests class as a convenience API.
- Annotations
- @Evolving() @Since( "3.1.0" )
- Source
- ExecutorResourceRequest.scala
- Alphabetic
- By Inheritance
- ExecutorResourceRequest
- Serializable
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
-
new
ExecutorResourceRequest(resourceName: String, amount: Long, discoveryScript: String = "", vendor: String = "")
- resourceName
Name of the resource
- amount
Amount requesting
- discoveryScript
Optional script used to discover the resources. This is required on some cluster managers that don't tell Spark the addresses of the resources allocated. The script runs on Executors startup to discover the addresses of the resources available.
- vendor
Optional vendor, required for some cluster managers
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- val amount: Long
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
- val discoveryScript: String
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(obj: Any): Boolean
- Definition Classes
- ExecutorResourceRequest → AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- ExecutorResourceRequest → AnyRef → Any
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- val resourceName: String
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- ExecutorResourceRequest → AnyRef → Any
- val vendor: String
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()