@Private
public interface ShuffleDriverComponents
Modifier and Type | Method and Description |
---|---|
void |
cleanupApplication()
Called once at the end of the Spark application to clean up any existing shuffle state.
|
java.util.Map<String,String> |
initializeApplication()
Called once in the driver to bootstrap this module that is specific to this application.
|
default void |
registerShuffle(int shuffleId)
Called once per shuffle id when the shuffle id is first generated for a shuffle stage.
|
default void |
removeShuffle(int shuffleId,
boolean blocking)
Removes shuffle data associated with the given shuffle.
|
default boolean |
supportsReliableStorage()
Does this shuffle component support reliable storage - external to the lifecycle of the
executor host ? For example, writing shuffle data to a distributed filesystem or
persisting it in a remote shuffle service.
|
java.util.Map<String,String> initializeApplication()
void cleanupApplication()
default void registerShuffle(int shuffleId)
shuffleId
- The unique identifier for the shuffle stage.default void removeShuffle(int shuffleId, boolean blocking)
shuffleId
- The unique identifier for the shuffle stage.blocking
- Whether this call should block on the deletion of the data.default boolean supportsReliableStorage()