Package org.apache.spark.sql.pipelines.graph

Class TriggeredGraphExecution

Object

org.apache.spark.sql.pipelines.graph.GraphExecution

org.apache.spark.sql.pipelines.graph.TriggeredGraphExecution

All Implemented Interfaces:

org.apache.spark.internal.Logging

public class TriggeredGraphExecution
extends GraphExecution

Executes all of the flows in the given graph in topological order. Each flow processes all available data before downstream flows are triggered.

param: graphForExecution the graph to execute. param: env the context in which the graph is executed. param: onCompletion a callback to execute after all streams are done. The boolean argument is true if the execution was successful. param: clock a clock used to determine the time of execution.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static interface	TriggeredGraphExecution.StreamState
static class	TriggeredGraphExecution.StreamState$

Nested classes/interfaces inherited from class org.apache.spark.sql.pipelines.graph.GraphExecution

GraphExecution.FlowExecutionAction, GraphExecution.FlowExecutionStopReason, GraphExecution.RetryFlowExecution$, GraphExecution.StopFlowExecution, GraphExecution.StopFlowExecution$

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
TriggeredGraphExecution(DataflowGraph graphForExecution, PipelineUpdateContext env, scala.Function1<RunTerminationReason,scala.runtime.BoxedUnit> onCompletion, Clock clock)

Method Summary

Modifier and Type	Method	Description
void	awaitCompletion()	Blocks the current thread while any flows are queued or running.
RunTerminationReason	getRunTerminationReason()	Returns the reason why this flow execution has terminated.
void	start()	Starts the execution of flows in graphForExecution.
void	stop()	Stops this execution by stopping all streams and terminating any other resources.
Trigger	streamTrigger(Flow flow)	The `Trigger` configuration for a streaming flow.

Methods inherited from class org.apache.spark.sql.pipelines.graph.GraphExecution

determineFlowExecutionActionFromError, flowExecutions, graphForExecution, LogStringContext, maxRetryAttemptsForFlow, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, planAndStartFlow, stopFlow, stopThread

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Constructor Details

TriggeredGraphExecution

public TriggeredGraphExecution(DataflowGraph graphForExecution,
 PipelineUpdateContext env,
 scala.Function1<RunTerminationReason,scala.runtime.BoxedUnit> onCompletion,
 Clock clock)

Method Details

streamTrigger

public Trigger streamTrigger(Flow flow)

Description copied from class: GraphExecution

The `Trigger` configuration for a streaming flow.

Specified by:

streamTrigger in class GraphExecution

start

public void start()

Description copied from class: GraphExecution

Starts the execution of flows in graphForExecution. Does not block.

Overrides:

start in class GraphExecution

awaitCompletion

public void awaitCompletion()

Description copied from class: GraphExecution

Blocks the current thread while any flows are queued or running. Returns when all flows that could be run have completed. When this returns, all flows are either SUCCESSFUL, TERMINATED_WITH_ERROR, SKIPPED, CANCELED, or EXCLUDED.

Specified by:

awaitCompletion in class GraphExecution

stop

public void stop()

Description copied from class: GraphExecution

Stops this execution by stopping all streams and terminating any other resources.

This method may be called multiple times due to race conditions and must be idempotent.

Overrides:

stop in class GraphExecution

getRunTerminationReason

public RunTerminationReason getRunTerminationReason()

Description copied from class: GraphExecution

Returns the reason why this flow execution has terminated. If the function is called before the flow has not terminated yet, the behavior is undefined, and may return UnexpectedRunFailure.

Specified by:

getRunTerminationReason in class GraphExecution

Returns:

(undocumented)