pyspark.sql.streaming.StreamingQueryManager.addListener¶

StreamingQueryManager.addListener(listener: pyspark.sql.streaming.listener.StreamingQueryListener) → None[source]¶

New in version 3.4.0.

Changed in version 3.5.0: Supports Spark Connect.

Parameters

listenerStreamingQueryListener: A StreamingQueryListener to receive up-calls for life cycle events of StreamingQuery.

Notes

This function behaves differently in Spark Connect mode. In Connect, the provided functions doesn’t have access to variables defined outside of it. Also in Connect, you need to use self.spark to access spark session. Using spark would throw an exception. In short, if you want to use spark session inside the listener, please use self.spark in Connect mode, and use spark otherwise.

Examples

>>> from pyspark.sql.streaming import StreamingQueryListener
>>> class TestListener(StreamingQueryListener):
...     def onQueryStarted(self, event):
...         pass
...
...     def onQueryProgress(self, event):
...         pass
...
...     def onQueryIdle(self, event):
...         pass
...
...     def onQueryTerminated(self, event):
...         pass
...
>>> test_listener = TestListener()

>>> spark.streams.addListener(test_listener)

Deregister streaming query listener

>>> spark.streams.removeListener(test_listener)

pyspark.sql.streaming.StreamingQueryManager.active

pyspark.sql.streaming.StreamingQueryManager.awaitAnyTermination