Class PipelineUpdateContextImpl
Object
org.apache.spark.sql.pipelines.graph.PipelineUpdateContextImpl
- All Implemented Interfaces:
PipelineUpdateContext
An implementation of the PipelineUpdateContext trait used in production.
param: unresolvedGraph The graph (unresolved) to be executed in this update.
param: eventCallback A callback function to be called when an event is added to the event buffer.
-
Constructor Summary
ConstructorsConstructorDescriptionPipelineUpdateContextImpl
(DataflowGraph unresolvedGraph, scala.Function1<PipelineEvent, scala.runtime.BoxedUnit> eventCallback) -
Method Summary
Modifier and TypeMethodDescriptionBuffer containing internal events that are emitted during a run of a pipeline.Emits internal flow progress events into the event buffer.Filter for which tables should be full refreshed when performing this update.Defines operations relates to end to end execution of a `DataflowGraph`.Filter for which tables should be refreshed when performing this update.org.apache.spark.sql.classic.SparkSession
spark()
The SparkSession for this update.The unresolved graph for this update.Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.spark.sql.pipelines.graph.PipelineUpdateContext
refreshFlows
-
Constructor Details
-
PipelineUpdateContextImpl
public PipelineUpdateContextImpl(DataflowGraph unresolvedGraph, scala.Function1<PipelineEvent, scala.runtime.BoxedUnit> eventCallback)
-
-
Method Details
-
eventBuffer
Description copied from interface:PipelineUpdateContext
Buffer containing internal events that are emitted during a run of a pipeline.- Specified by:
eventBuffer
in interfacePipelineUpdateContext
-
flowProgressEventLogger
Description copied from interface:PipelineUpdateContext
Emits internal flow progress events into the event buffer.- Specified by:
flowProgressEventLogger
in interfacePipelineUpdateContext
-
fullRefreshTables
Description copied from interface:PipelineUpdateContext
Filter for which tables should be full refreshed when performing this update.- Specified by:
fullRefreshTables
in interfacePipelineUpdateContext
-
pipelineExecution
Description copied from interface:PipelineUpdateContext
Defines operations relates to end to end execution of a `DataflowGraph`.- Specified by:
pipelineExecution
in interfacePipelineUpdateContext
-
refreshTables
Description copied from interface:PipelineUpdateContext
Filter for which tables should be refreshed when performing this update.- Specified by:
refreshTables
in interfacePipelineUpdateContext
-
resetCheckpointFlows
- Specified by:
resetCheckpointFlows
in interfacePipelineUpdateContext
-
spark
public org.apache.spark.sql.classic.SparkSession spark()Description copied from interface:PipelineUpdateContext
The SparkSession for this update.- Specified by:
spark
in interfacePipelineUpdateContext
-
unresolvedGraph
Description copied from interface:PipelineUpdateContext
The unresolved graph for this update.- Specified by:
unresolvedGraph
in interfacePipelineUpdateContext
-