public class OutputCommitCoordinator extends Object implements Logging
OutputCommitCoordinator is instantiated in both the drivers and executors. On executors, it is configured with a reference to the driver's OutputCommitCoordinatorActor, so requests to commit output will be forwarded to the driver's OutputCommitCoordinator.
This class was introduced in SPARK-4879; see that JIRA issue (and the associated pull requests) for an extensive design discussion.
Modifier and Type | Class and Description |
---|---|
static class |
OutputCommitCoordinator.OutputCommitCoordinatorActor |
Constructor and Description |
---|
OutputCommitCoordinator(SparkConf conf) |
Modifier and Type | Method and Description |
---|---|
boolean |
canCommit(int stage,
long partition,
long attempt)
Called by tasks to ask whether they can commit their output to HDFS.
|
scala.Option<akka.actor.ActorRef> |
coordinatorActor() |
boolean |
handleAskPermissionToCommit(int stage,
long partition,
long attempt) |
boolean |
isEmpty()
Returns whether the OutputCommitCoordinator's internal data structures are all empty.
|
void |
stageEnd(int stage) |
void |
stageStart(int stage) |
void |
stop() |
void |
taskCompleted(int stage,
long partition,
long attempt,
TaskEndReason reason) |
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
public OutputCommitCoordinator(SparkConf conf)
public scala.Option<akka.actor.ActorRef> coordinatorActor()
public boolean isEmpty()
public boolean canCommit(int stage, long partition, long attempt)
If a task attempt has been authorized to commit, then all other attempts to commit the same task will be denied. If the authorized task attempt fails (e.g. due to its executor being lost), then a subsequent task attempt may be authorized to commit its output.
stage
- the stage numberpartition
- the partition numberattempt
- a unique identifier for this task attemptpublic void stageStart(int stage)
public void stageEnd(int stage)
public void taskCompleted(int stage, long partition, long attempt, TaskEndReason reason)
public void stop()
public boolean handleAskPermissionToCommit(int stage, long partition, long attempt)