pyspark.sql.streaming.StreamingQueryManager.get

StreamingQueryManager.get(id: str) → pyspark.sql.streaming.query.StreamingQuery[source]

Returns an active query from this SparkSession.

New in version 2.0.0.

Parameters
idstr

The unique id of specified query.

Returns
StreamingQuery

An active query with id from this SparkSession.

Notes

Exception will be thrown if an active query with this id does not exist.

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sdf.printSchema()
root
  |-- timestamp: timestamp (nullable = true)
  |-- value: long (nullable = true)
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()
>>> sq.name
'this_query'

Get an active query by id

>>> sq = spark.streams.get(sq.id)
>>> sq.isActive
True
>>> sq.stop()