Packages

t

org.apache.spark.sql.pipelines.graph

StreamingFlowExecution

trait StreamingFlowExecution extends FlowExecution with Logging

A 'FlowExecution' that processes data statefully using Structured Streaming.

Source
FlowExecution.scala
Linear Supertypes
Logging, FlowExecution, AnyRef, Any
Known Subclasses
SinkWrite, StreamingTableWrite
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingFlowExecution
  2. Logging
  3. FlowExecution
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Abstract Value Members

  1. abstract def checkpointPath: String

    Structured Streaming checkpoint.

  2. abstract def destination: Output

    The destination that this FlowExecution is writing to.

    The destination that this FlowExecution is writing to.

    Definition Classes
    FlowExecution
  3. abstract def flow: ResolvedFlow

    The ResolvedFlow that this StreamingFlowExecution is executing.

  4. abstract def getOrigin: QueryOrigin

    Origin to use when recording events for this flow.

    Origin to use when recording events for this flow.

    Definition Classes
    FlowExecution
  5. abstract def identifier: TableIdentifier

    Identifier of this physical flow

    Identifier of this physical flow

    Definition Classes
    FlowExecution
  6. abstract def sqlConf: Map[String, String]

    Spark confs that must be set when starting this flow.

    Spark confs that must be set when starting this flow.

    Attributes
    protected
  7. abstract def startStream(): StreamingQuery

    Starts a stream and returns its streaming query.

    Starts a stream and returns its streaming query.

    Attributes
    protected
  8. abstract def trigger: Trigger

    Structured Streaming trigger.

  9. abstract def updateContext: PipelineUpdateContext

    Context about this pipeline update.

    Context about this pipeline update.

    Definition Classes
    FlowExecution

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. final def displayName: String

    Returns a user-visible name for the flow.

    Returns a user-visible name for the flow.

    Definition Classes
    FlowExecution
  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def exception: Option[Throwable]

    Returns an optional exception that occurred during execution, if any.

    Returns an optional exception that occurred during execution, if any.

    Definition Classes
    FlowExecution
  11. final def executeAsync(): Unit

    Executes this FlowExecution asynchronously to perform its intended update.

    Executes this FlowExecution asynchronously to perform its intended update. A future that can be used to track execution status is saved, and can be retrieved with getFuture.

    Definition Classes
    FlowExecution
  12. final def executeInternal(): Future[Unit]

    Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.

    Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.

    returns

    a Future that completes when the execution is finished or stopped.

    Definition Classes
    StreamingFlowExecution → FlowExecution
  13. implicit val executionContext: ExecutionContext

    The thread execution context for the current FlowExecution.

    The thread execution context for the current FlowExecution.

    Definition Classes
    FlowExecution
  14. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  15. def getFuture: Future[ExecutionResult]

    Retrieves the future that can be used to track execution status.

    Retrieves the future that can be used to track execution status.

    Definition Classes
    FlowExecution
  16. def getStreamingQuery: StreamingQuery

    Visible for testing

  17. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  18. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def isCompleted: Boolean

    Returns true if and only if this FlowExecution has been completed with either success or an exception.

    Returns true if and only if this FlowExecution has been completed with either success or an exception.

    Definition Classes
    FlowExecution
  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. def isStreaming: Boolean

    Returns true iff this FlowExecution executes using Spark Structured Streaming.

    Returns true iff this FlowExecution executes using Spark Structured Streaming.

    Definition Classes
    StreamingFlowExecution → FlowExecution
  23. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  24. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  25. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  30. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  39. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  45. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  48. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  49. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  50. def spark: classic.SparkSession

    SparkSession to execute this physical flow with.

    SparkSession to execute this physical flow with.

    The default value for streaming flows is the pipeline's spark session because the source dataframe is resolved using the pipeline's spark session, and a new session will be started implicitly by the streaming query.

    The default value for batch flows is a cloned spark session from the pipeline's spark session.

    Please make sure that the execution thread runs in a different spark session than the pipeline's spark session.

    Attributes
    protected
    Definition Classes
    FlowExecution
  51. def stop(): Unit

    Stops execution of this FlowExecution.

    Stops execution of this FlowExecution. If you override this, please be sure to call super.stop() at the beginning of your method, so we can properly handle errors when a user tries to stop a flow.

    Definition Classes
    FlowExecution
  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def toString(): String
    Definition Classes
    AnyRef → Any
  54. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  55. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  56. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  57. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from FlowExecution

Inherited from AnyRef

Inherited from Any

Ungrouped