Query Management#

StreamingQuery.awaitTermination([timeout])

Waits for the termination of this query, either by query.stop() or by an exception.

StreamingQuery.exception()

New in version 2.1.0.

StreamingQuery.explain([extended])

Prints the (logical and physical) plans to the console for debugging purpose.

StreamingQuery.id

Returns the unique id of this query that persists across restarts from checkpoint data.

StreamingQuery.isActive

Whether this streaming query is currently active or not.

StreamingQuery.lastProgress

Returns the most recent StreamingQueryProgress update of this streaming query or None if there were no progress updates

StreamingQuery.name

Returns the user-specified name of the query, or None if not specified.

StreamingQuery.processAllAvailable()

Blocks until all available data in the source has been processed and committed to the sink.

StreamingQuery.recentProgress

Returns an array of the most recent [[StreamingQueryProgress]] updates for this query.

StreamingQuery.runId

Returns the unique id of this query that does not persist across restarts.

StreamingQuery.status

Returns the current status of the query.

StreamingQuery.stop()

Stop this streaming query.

StreamingQueryManager.active

Returns a list of active queries associated with this SparkSession

StreamingQueryManager.addListener(listener)

Register a StreamingQueryListener to receive up-calls for life cycle events of StreamingQuery.

StreamingQueryManager.awaitAnyTermination([...])

Wait until any of the queries on the associated SparkSession has terminated since the creation of the context, or since resetTerminated() was called.

StreamingQueryManager.get(id)

Returns an active query from this SparkSession.

StreamingQueryManager.removeListener(listener)

Deregister a StreamingQueryListener.

StreamingQueryManager.resetTerminated()

Forget about past terminated queries so that awaitAnyTermination() can be used again to wait for new terminations.