Package org.apache.spark.sql.pipelines.graph

Class StreamingTableWrite

Object

org.apache.spark.sql.pipelines.graph.StreamingTableWrite

All Implemented Interfaces:

org.apache.spark.internal.Logging, FlowExecution, StreamingFlowExecution

public class StreamingTableWrite
extends Object
implements StreamingFlowExecution

A `StreamingFlowExecution` that writes a streaming `DataFrame` to a `Table`.

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
StreamingTableWrite(org.apache.spark.sql.catalyst.TableIdentifier identifier, ResolvedFlow flow, DataflowGraph graph, PipelineUpdateContext updateContext, String checkpointPath, Trigger trigger, Table destination, scala.collection.immutable.Map<String,String> sqlConf)

Method Summary

Modifier and Type	Method	Description
String	checkpointPath()	Structured Streaming checkpoint.
Table	destination()	The destination that this `FlowExecution` is writing to.
scala.concurrent.ExecutionContext	executionContext()	The thread execution context for the current `FlowExecution`.
ResolvedFlow	flow()	The `ResolvedFlow` that this `StreamingFlowExecution` is executing.
QueryOrigin	getOrigin()	Origin to use when recording events for this flow.
DataflowGraph	graph()
org.apache.spark.sql.catalyst.TableIdentifier	identifier()	Identifier of this physical flow
scala.collection.immutable.Map<String,String>	sqlConf()	Spark confs that must be set when starting this flow.
StreamingQuery	startStream()	Starts a stream and returns its streaming query.
Trigger	trigger()	Structured Streaming trigger.
PipelineUpdateContext	updateContext()	Context about this pipeline update.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.sql.pipelines.graph.FlowExecution

displayName, exception, executeAsync, getFuture, isCompleted, spark, stop

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Methods inherited from interface org.apache.spark.sql.pipelines.graph.StreamingFlowExecution

executeInternal, getStreamingQuery, isStreaming

Constructor Details

StreamingTableWrite

public StreamingTableWrite(org.apache.spark.sql.catalyst.TableIdentifier identifier,
 ResolvedFlow flow,
 DataflowGraph graph,
 PipelineUpdateContext updateContext,
 String checkpointPath,
 Trigger trigger,
 Table destination,
 scala.collection.immutable.Map<String,String> sqlConf)

Method Details

checkpointPath

public String checkpointPath()

Description copied from interface: StreamingFlowExecution

Structured Streaming checkpoint.

Specified by:

checkpointPath in interface StreamingFlowExecution

destination

public Table destination()

Description copied from interface: FlowExecution

The destination that this `FlowExecution` is writing to.

Specified by:

destination in interface FlowExecution

executionContext

public scala.concurrent.ExecutionContext executionContext()

Description copied from interface: FlowExecution

The thread execution context for the current `FlowExecution`.

Specified by:

executionContext in interface FlowExecution

flow

public ResolvedFlow flow()

Description copied from interface: StreamingFlowExecution

The `ResolvedFlow` that this `StreamingFlowExecution` is executing.

Specified by:

flow in interface StreamingFlowExecution

getOrigin

public QueryOrigin getOrigin()

Description copied from interface: FlowExecution

Origin to use when recording events for this flow.

Specified by:

getOrigin in interface FlowExecution

Returns:

(undocumented)

graph

public DataflowGraph graph()

identifier

public org.apache.spark.sql.catalyst.TableIdentifier identifier()

Description copied from interface: FlowExecution

Identifier of this physical flow

Specified by:

identifier in interface FlowExecution

sqlConf

public scala.collection.immutable.Map<String,String> sqlConf()

Description copied from interface: StreamingFlowExecution

Spark confs that must be set when starting this flow.

Specified by:

sqlConf in interface StreamingFlowExecution

startStream

public StreamingQuery startStream()

Description copied from interface: StreamingFlowExecution

Starts a stream and returns its streaming query.

Specified by:

startStream in interface StreamingFlowExecution

trigger

public Trigger trigger()

Description copied from interface: StreamingFlowExecution

Structured Streaming trigger.

Specified by:

trigger in interface StreamingFlowExecution

updateContext

public PipelineUpdateContext updateContext()

Description copied from interface: FlowExecution

Context about this pipeline update.

Specified by:

updateContext in interface FlowExecution