Package org.apache.spark.sql.pipelines.graph

Interface FlowExecution

All Known Subinterfaces:
StreamingFlowExecution
All Known Implementing Classes:
BatchTableWrite, StreamingTableWrite
public interface FlowExecution
A `FlowExecution` specifies how to execute a flow and manages its execution.

  • Method Summary

    Modifier and Type
    Method
    Description
    Output
    destination()
    The destination that this `FlowExecution` is writing to.
    String
    displayName()
    Returns a user-visible name for the flow.
    scala.Option<Throwable>
    exception()
    Returns an optional exception that occurred during execution, if any.
    void
    executeAsync()
    Executes this FlowExecution asynchronously to perform its intended update.
    scala.concurrent.Future<scala.runtime.BoxedUnit>
    executeInternal()
    Executes this FlowExecution synchronously to perform its intended update.
    scala.concurrent.ExecutionContext
    executionContext()
    The thread execution context for the current `FlowExecution`.
    scala.concurrent.Future<ExecutionResult>
    getFuture()
    Retrieves the future that can be used to track execution status.
    QueryOrigin
    getOrigin()
    Origin to use when recording events for this flow.
    org.apache.spark.sql.catalyst.TableIdentifier
    identifier()
    Identifier of this physical flow
    boolean
    isCompleted()
    Returns true if and only if this FlowExecution has been completed with either success or an exception.
    boolean
    isStreaming()
    Returns true iff this `FlowExecution` executes using Spark Structured Streaming.
    org.apache.spark.sql.classic.SparkSession
    spark()
    SparkSession to execute this physical flow with.
    void
    stop()
    Stops execution of this FlowExecution.
    PipelineUpdateContext
    updateContext()
    Context about this pipeline update.

  • Method Details

    • identifier

      org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Identifier of this physical flow

    • displayName

      String displayName()
      Returns a user-visible name for the flow.
      Returns:
      (undocumented)

    • spark

      org.apache.spark.sql.classic.SparkSession spark()
      SparkSession to execute this physical flow with.

      The default value for streaming flows is the pipeline's spark session because the source dataframe is resolved using the pipeline's spark session, and a new session will be started implicitly by the streaming query.

      The default value for batch flows is a cloned spark session from the pipeline's spark session.

      Please make sure that the execution thread runs in a different spark session than the pipeline's spark session.

      Returns:
      (undocumented)

    • getOrigin

      QueryOrigin getOrigin()
      Origin to use when recording events for this flow.
      Returns:
      (undocumented)

    • isCompleted

      boolean isCompleted()
      Returns true if and only if this FlowExecution has been completed with either success or an exception.
      Returns:
      (undocumented)

    • isStreaming

      boolean isStreaming()
      Returns true iff this `FlowExecution` executes using Spark Structured Streaming.

    • getFuture

      scala.concurrent.Future<ExecutionResult> getFuture()
      Retrieves the future that can be used to track execution status.

    • updateContext

      PipelineUpdateContext updateContext()
      Context about this pipeline update.

    • executionContext

      scala.concurrent.ExecutionContext executionContext()
      The thread execution context for the current `FlowExecution`.

    • stop

      void stop()
      Stops execution of this FlowExecution. If you override this, please be sure to call super.stop() at the beginning of your method, so we can properly handle errors when a user tries to stop a flow.

    • exception

      scala.Option<Throwable> exception()
      Returns an optional exception that occurred during execution, if any.

    • executeInternal

      scala.concurrent.Future<scala.runtime.BoxedUnit> executeInternal()
      Executes this FlowExecution synchronously to perform its intended update. This method should be overridden by subclasses to provide the actual execution logic.

      Returns:
      a Future that completes when the execution is finished or stopped.

    • executeAsync

      void executeAsync()
      Executes this FlowExecution asynchronously to perform its intended update. A future that can be used to track execution status is saved, and can be retrieved with getFuture.

    • destination

      Output destination()
      The destination that this `FlowExecution` is writing to.