trait ShuffleDriverComponents extends AnyRef
:: Private :: An interface for building shuffle support modules for the Driver.
- Annotations
- @Private()
- Source
- ShuffleDriverComponents.java
- Alphabetic
- By Inheritance
- ShuffleDriverComponents
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def cleanupApplication(): Unit
Called once at the end of the Spark application to clean up any existing shuffle state.
- abstract def initializeApplication(): Map[String, String]
Called once in the driver to bootstrap this module that is specific to this application.
Called once in the driver to bootstrap this module that is specific to this application. This method is called before submitting executor requests to the cluster manager.
This method should prepare the module with its shuffle components i.e. registering against an external file servers or shuffle services, or creating tables in a shuffle storage data database.
- returns
additional SparkConf settings necessary for initializing the executor components. This would include configurations that cannot be statically set on the application, like the host:port of external services for shuffle storage.
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def registerShuffle(shuffleId: Int): Unit
Called once per shuffle id when the shuffle id is first generated for a shuffle stage.
Called once per shuffle id when the shuffle id is first generated for a shuffle stage.
- shuffleId
The unique identifier for the shuffle stage.
- def removeShuffle(shuffleId: Int, blocking: Boolean): Unit
Removes shuffle data associated with the given shuffle.
Removes shuffle data associated with the given shuffle.
- shuffleId
The unique identifier for the shuffle stage.
- blocking
Whether this call should block on the deletion of the data.
- def supportsReliableStorage(): Boolean
Does this shuffle component support reliable storage - external to the lifecycle of the executor host ? For example, writing shuffle data to a distributed filesystem or persisting it in a remote shuffle service.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)