Package org.apache.spark.sql.pipelines.graph

Interface PipelineUpdateContext

All Known Implementing Classes:
PipelineUpdateContextImpl
public interface PipelineUpdateContext

  • Method Details

    • eventCallback

      scala.Function1<PipelineEvent,scala.runtime.BoxedUnit> eventCallback()
      Callback to invoke for internal events that are emitted during a run of a pipeline.

    • flowProgressEventLogger

      FlowProgressEventLogger flowProgressEventLogger()
      Emits internal flow progress events into the event buffer.

    • fullRefreshTables

      TableFilter fullRefreshTables()
      Filter for which tables should be full refreshed when performing this update.

    • pipelineExecution

      PipelineExecution pipelineExecution()
      Defines operations relates to end to end execution of a `DataflowGraph`.

    • refreshFlows

      FlowFilter refreshFlows()
      Filter for which flows should be refreshed when performing this update. Should be a superset of fullRefreshFlows.
      Returns:
      (undocumented)

    • refreshTables

      TableFilter refreshTables()
      Filter for which tables should be refreshed when performing this update.

    • resetCheckpointFlows

      FlowFilter resetCheckpointFlows()

    • spark

      org.apache.spark.sql.classic.SparkSession spark()
      The SparkSession for this update.

    • storageRoot

      String storageRoot()
      The root storage location for pipeline metadata, including checkpoints for streaming flows.
      Returns:
      (undocumented)

    • unresolvedGraph

      DataflowGraph unresolvedGraph()
      The unresolved graph for this update.