Package org.apache.spark.sql.pipelines.graph

Interface StreamingFlowExecution

All Superinterfaces:
FlowExecution, org.apache.spark.internal.Logging
All Known Implementing Classes:
SinkWrite, StreamingTableWrite
public interface StreamingFlowExecution extends FlowExecution, org.apache.spark.internal.Logging
A 'FlowExecution' that processes data statefully using Structured Streaming.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    checkpointPath()
    Structured Streaming checkpoint.
    scala.concurrent.Future<scala.runtime.BoxedUnit>
    executeInternal()
    Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.
    ResolvedFlow
    flow()
    The `ResolvedFlow` that this `StreamingFlowExecution` is executing.
    StreamingQuery
    getStreamingQuery()
    Visible for testing
    boolean
    isStreaming()
    Returns true iff this `FlowExecution` executes using Spark Structured Streaming.
    scala.collection.immutable.Map<String,String>
    sqlConf()
    Spark confs that must be set when starting this flow.
    StreamingQuery
    startStream()
    Starts a stream and returns its streaming query.
    Trigger
    trigger()
    Structured Streaming trigger.

    Methods inherited from interface org.apache.spark.sql.pipelines.graph.FlowExecution

    destination, displayName, exception, executeAsync, executionContext, getFuture, getOrigin, identifier, isCompleted, spark, stop, updateContext

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Method Details

    • checkpointPath

      String checkpointPath()
      Structured Streaming checkpoint.

    • executeInternal

      scala.concurrent.Future<scala.runtime.BoxedUnit> executeInternal()
      Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.
      Specified by:
      executeInternal in interface FlowExecution
      Returns:
      (undocumented)

    • flow

      ResolvedFlow flow()
      The `ResolvedFlow` that this `StreamingFlowExecution` is executing.

    • getStreamingQuery

      StreamingQuery getStreamingQuery()
      Visible for testing

    • isStreaming

      boolean isStreaming()
      Description copied from interface: FlowExecution
      Returns true iff this `FlowExecution` executes using Spark Structured Streaming.
      Specified by:
      isStreaming in interface FlowExecution

    • sqlConf

      scala.collection.immutable.Map<String,String> sqlConf()
      Spark confs that must be set when starting this flow.

    • startStream

      StreamingQuery startStream()
      Starts a stream and returns its streaming query.

    • trigger

      Trigger trigger()
      Structured Streaming trigger.