pyspark.sql.streaming.StreamingQueryManager.get#

StreamingQueryManager.get(id)[source]#

Returns an active query from this SparkSession.

Added in version 2.0.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters:

idstr: The unique id of specified query.

Returns:

StreamingQuery: An active query with id from this SparkSession.

Notes

Exception will be thrown if an active query with this id does not exist.

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sdf.printSchema()
root
  |-- timestamp: timestamp (nullable = true)
  |-- value: long (nullable = true)

>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sq.name
'this_query'

Get an active query by id

>>> sq = spark.streams.get(sq.id)
>>> sq.isActive
True
>>> sq.stop()