Class StreamingTableWrite
Object
org.apache.spark.sql.pipelines.graph.StreamingTableWrite
- All Implemented Interfaces:
org.apache.spark.internal.Logging
,FlowExecution
,StreamingFlowExecution
A `StreamingFlowExecution` that writes a streaming `DataFrame` to a `Table`.
-
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging
org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter
-
Constructor Summary
ConstructorsConstructorDescriptionStreamingTableWrite
(org.apache.spark.sql.catalyst.TableIdentifier identifier, ResolvedFlow flow, DataflowGraph graph, PipelineUpdateContext updateContext, String checkpointPath, Trigger trigger, Table destination, scala.collection.immutable.Map<String, String> sqlConf) -
Method Summary
Modifier and TypeMethodDescriptionStructured Streaming checkpoint.The destination that this `FlowExecution` is writing to.scala.concurrent.ExecutionContext
The thread execution context for the current `FlowExecution`.flow()
The `ResolvedFlow` that this `StreamingFlowExecution` is executing.Origin to use when recording events for this flow.graph()
org.apache.spark.sql.catalyst.TableIdentifier
Identifier of this physical flowsqlConf()
Spark confs that must be set when starting this flow.Starts a stream and returns its streaming query.trigger()
Structured Streaming trigger.Context about this pipeline update.Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.spark.sql.pipelines.graph.FlowExecution
displayName, exception, executeAsync, getFuture, isCompleted, spark, stop
Methods inherited from interface org.apache.spark.internal.Logging
initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext
Methods inherited from interface org.apache.spark.sql.pipelines.graph.StreamingFlowExecution
executeInternal, isStreaming
-
Constructor Details
-
StreamingTableWrite
public StreamingTableWrite(org.apache.spark.sql.catalyst.TableIdentifier identifier, ResolvedFlow flow, DataflowGraph graph, PipelineUpdateContext updateContext, String checkpointPath, Trigger trigger, Table destination, scala.collection.immutable.Map<String, String> sqlConf)
-
-
Method Details
-
checkpointPath
Description copied from interface:StreamingFlowExecution
Structured Streaming checkpoint.- Specified by:
checkpointPath
in interfaceStreamingFlowExecution
-
destination
Description copied from interface:FlowExecution
The destination that this `FlowExecution` is writing to.- Specified by:
destination
in interfaceFlowExecution
-
executionContext
public scala.concurrent.ExecutionContext executionContext()Description copied from interface:FlowExecution
The thread execution context for the current `FlowExecution`.- Specified by:
executionContext
in interfaceFlowExecution
-
flow
Description copied from interface:StreamingFlowExecution
The `ResolvedFlow` that this `StreamingFlowExecution` is executing.- Specified by:
flow
in interfaceStreamingFlowExecution
-
getOrigin
Description copied from interface:FlowExecution
Origin to use when recording events for this flow.- Specified by:
getOrigin
in interfaceFlowExecution
- Returns:
- (undocumented)
-
graph
-
identifier
public org.apache.spark.sql.catalyst.TableIdentifier identifier()Description copied from interface:FlowExecution
Identifier of this physical flow- Specified by:
identifier
in interfaceFlowExecution
-
sqlConf
Description copied from interface:StreamingFlowExecution
Spark confs that must be set when starting this flow.- Specified by:
sqlConf
in interfaceStreamingFlowExecution
-
startStream
Description copied from interface:StreamingFlowExecution
Starts a stream and returns its streaming query.- Specified by:
startStream
in interfaceStreamingFlowExecution
-
trigger
Description copied from interface:StreamingFlowExecution
Structured Streaming trigger.- Specified by:
trigger
in interfaceStreamingFlowExecution
-
updateContext
Description copied from interface:FlowExecution
Context about this pipeline update.- Specified by:
updateContext
in interfaceFlowExecution
-