Packages

c

org.apache.spark.sql.pipelines.graph

StreamingTableWrite

class StreamingTableWrite extends StreamingFlowExecution

A StreamingFlowExecution that writes a streaming DataFrame to a Table.

Source
FlowExecution.scala
Linear Supertypes
StreamingFlowExecution, Logging, FlowExecution, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingTableWrite
  2. StreamingFlowExecution
  3. Logging
  4. FlowExecution
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new StreamingTableWrite(identifier: TableIdentifier, flow: ResolvedFlow, graph: DataflowGraph, updateContext: PipelineUpdateContext, checkpointPath: String, trigger: Trigger, destination: Table, sqlConf: Map[String, String])

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. val checkpointPath: String

    Structured Streaming checkpoint.

    Structured Streaming checkpoint.

    Definition Classes
    StreamingTableWrite → StreamingFlowExecution
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. val destination: Table

    The destination that this FlowExecution is writing to.

    The destination that this FlowExecution is writing to.

    Definition Classes
    StreamingTableWrite → FlowExecution
  9. final def displayName: String

    Returns a user-visible name for the flow.

    Returns a user-visible name for the flow.

    Definition Classes
    FlowExecution
  10. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  12. def exception: Option[Throwable]

    Returns an optional exception that occurred during execution, if any.

    Returns an optional exception that occurred during execution, if any.

    Definition Classes
    FlowExecution
  13. final def executeAsync(): Unit

    Executes this FlowExecution asynchronously to perform its intended update.

    Executes this FlowExecution asynchronously to perform its intended update. A future that can be used to track execution status is saved, and can be retrieved with getFuture.

    Definition Classes
    FlowExecution
  14. final def executeInternal(): Future[Unit]

    Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.

    Executes this StreamingFlowExecution by starting its stream with the correct scheduling pool and confs.

    returns

    a Future that completes when the execution is finished or stopped.

    Definition Classes
    StreamingFlowExecution → FlowExecution
  15. implicit val executionContext: ExecutionContext

    The thread execution context for the current FlowExecution.

    The thread execution context for the current FlowExecution.

    Definition Classes
    FlowExecution
  16. val flow: ResolvedFlow

    The ResolvedFlow that this StreamingFlowExecution is executing.

    The ResolvedFlow that this StreamingFlowExecution is executing.

    Definition Classes
    StreamingTableWrite → StreamingFlowExecution
  17. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  18. def getFuture: Future[ExecutionResult]

    Retrieves the future that can be used to track execution status.

    Retrieves the future that can be used to track execution status.

    Definition Classes
    FlowExecution
  19. def getOrigin: QueryOrigin

    Origin to use when recording events for this flow.

    Origin to use when recording events for this flow.

    Definition Classes
    StreamingTableWrite → FlowExecution
  20. def getStreamingQuery: StreamingQuery

    Visible for testing

    Visible for testing

    Definition Classes
    StreamingFlowExecution
  21. val graph: DataflowGraph
  22. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  23. val identifier: TableIdentifier

    Identifier of this physical flow

    Identifier of this physical flow

    Definition Classes
    StreamingTableWrite → FlowExecution
  24. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  25. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def isCompleted: Boolean

    Returns true if and only if this FlowExecution has been completed with either success or an exception.

    Returns true if and only if this FlowExecution has been completed with either success or an exception.

    Definition Classes
    FlowExecution
  27. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  28. def isStreaming: Boolean

    Returns true iff this FlowExecution executes using Spark Structured Streaming.

    Returns true iff this FlowExecution executes using Spark Structured Streaming.

    Definition Classes
    StreamingFlowExecution → FlowExecution
  29. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  30. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  31. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  42. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  43. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  44. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  45. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  46. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  47. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  48. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  49. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  50. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  51. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  52. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  53. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  54. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  55. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  56. def spark: classic.SparkSession

    SparkSession to execute this physical flow with.

    SparkSession to execute this physical flow with.

    The default value for streaming flows is the pipeline's spark session because the source dataframe is resolved using the pipeline's spark session, and a new session will be started implicitly by the streaming query.

    The default value for batch flows is a cloned spark session from the pipeline's spark session.

    Please make sure that the execution thread runs in a different spark session than the pipeline's spark session.

    Attributes
    protected
    Definition Classes
    FlowExecution
  57. val sqlConf: Map[String, String]

    Spark confs that must be set when starting this flow.

    Spark confs that must be set when starting this flow.

    Definition Classes
    StreamingTableWrite → StreamingFlowExecution
  58. def startStream(): StreamingQuery

    Starts a stream and returns its streaming query.

    Starts a stream and returns its streaming query.

    Definition Classes
    StreamingTableWrite → StreamingFlowExecution
  59. def stop(): Unit

    Stops execution of this FlowExecution.

    Stops execution of this FlowExecution. If you override this, please be sure to call super.stop() at the beginning of your method, so we can properly handle errors when a user tries to stop a flow.

    Definition Classes
    FlowExecution
  60. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  61. def toString(): String
    Definition Classes
    AnyRef → Any
  62. val trigger: Trigger

    Structured Streaming trigger.

    Structured Streaming trigger.

    Definition Classes
    StreamingTableWrite → StreamingFlowExecution
  63. val updateContext: PipelineUpdateContext

    Context about this pipeline update.

    Context about this pipeline update.

    Definition Classes
    StreamingTableWrite → FlowExecution
  64. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  65. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  66. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  67. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from StreamingFlowExecution

Inherited from Logging

Inherited from FlowExecution

Inherited from AnyRef

Inherited from Any

Ungrouped