trait StreamingQuery extends AnyRef
A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.
- Annotations
- @Evolving()
- Source
- StreamingQuery.scala
- Since
2.0.0
- Alphabetic
- By Inheritance
- StreamingQuery
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Abstract Value Members
-
abstract
def
awaitTermination(timeoutMs: Long): Boolean
Waits for the termination of
this
query, either byquery.stop()
or by an exception.Waits for the termination of
this
query, either byquery.stop()
or by an exception. If the query has terminated with an exception, then the exception will be thrown. Otherwise, it returns whether the query has terminated or not within thetimeoutMs
milliseconds.If the query has terminated, then all subsequent calls to this method will either return
true
immediately (if the query was terminated bystop()
), or throw the exception immediately (if the query has terminated with exception).- Annotations
- @throws( ... )
- Since
2.0.0
- Exceptions thrown
StreamingQueryException
if the query has terminated with an exception
-
abstract
def
awaitTermination(): Unit
Waits for the termination of
this
query, either byquery.stop()
or by an exception.Waits for the termination of
this
query, either byquery.stop()
or by an exception. If the query has terminated with an exception, then the exception will be thrown.If the query has terminated, then all subsequent calls to this method will either return immediately (if the query was terminated by
stop()
), or throw the exception immediately (if the query has terminated with exception).- Annotations
- @throws( ... )
- Since
2.0.0
- Exceptions thrown
StreamingQueryException
if the query has terminated with an exception.
-
abstract
def
exception: Option[StreamingQueryException]
Returns the StreamingQueryException if the query was terminated by an exception.
Returns the StreamingQueryException if the query was terminated by an exception.
- Since
2.0.0
-
abstract
def
explain(extended: Boolean): Unit
Prints the physical plan to the console for debugging purposes.
Prints the physical plan to the console for debugging purposes.
- extended
whether to do extended explain or not
- Since
2.0.0
-
abstract
def
explain(): Unit
Prints the physical plan to the console for debugging purposes.
Prints the physical plan to the console for debugging purposes.
- Since
2.0.0
-
abstract
def
id: UUID
Returns the unique id of this query that persists across restarts from checkpoint data.
Returns the unique id of this query that persists across restarts from checkpoint data. That is, this id is generated when a query is started for the first time, and will be the same every time it is restarted from checkpoint data. Also see runId.
- Since
2.1.0
-
abstract
def
isActive: Boolean
Returns
true
if this query is actively running.Returns
true
if this query is actively running.- Since
2.0.0
-
abstract
def
lastProgress: StreamingQueryProgress
Returns the most recent StreamingQueryProgress update of this streaming query.
Returns the most recent StreamingQueryProgress update of this streaming query.
- Since
2.1.0
-
abstract
def
name: String
Returns the user-specified name of the query, or null if not specified.
Returns the user-specified name of the query, or null if not specified. This name can be specified in the
org.apache.spark.sql.streaming.DataStreamWriter
asdataframe.writeStream.queryName("query").start()
. This name, if set, must be unique across all active queries.- Since
2.0.0
-
abstract
def
processAllAvailable(): Unit
Blocks until all available data in the source has been processed and committed to the sink.
Blocks until all available data in the source has been processed and committed to the sink. This method is intended for testing. Note that in the case of continually arriving data, this method may block forever. Additionally, this method is only guaranteed to block until data that has been synchronously appended data to a
org.apache.spark.sql.execution.streaming.Source
prior to invocation. (i.e.getOffset
must immediately reflect the addition).- Since
2.0.0
-
abstract
def
recentProgress: Array[StreamingQueryProgress]
Returns an array of the most recent StreamingQueryProgress updates for this query.
Returns an array of the most recent StreamingQueryProgress updates for this query. The number of progress updates retained for each stream is configured by Spark session configuration
spark.sql.streaming.numRecentProgressUpdates
.- Since
2.1.0
-
abstract
def
runId: UUID
Returns the unique id of this run of the query.
-
abstract
def
sparkSession: SparkSession
Returns the
SparkSession
associated withthis
.Returns the
SparkSession
associated withthis
.- Since
2.0.0
-
abstract
def
status: StreamingQueryStatus
Returns the current status of the query.
Returns the current status of the query.
- Since
2.0.2
-
abstract
def
stop(): Unit
Stops the execution of this query if it is running.
Stops the execution of this query if it is running. This waits until the termination of the query execution threads or until a timeout is hit.
By default stop will block indefinitely. You can configure a timeout by the configuration
spark.sql.streaming.stopTimeout
. A timeout of 0 (or negative) milliseconds will block indefinitely. If aTimeoutException
is thrown, users can retry stopping the stream. If the issue persists, it is advisable to kill the Spark application.- Annotations
- @throws( ... )
- Since
2.0.0
Concrete Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- AnyRef → Any
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()