pyspark.sql.streaming.StreamingQuery.recentProgress¶

property StreamingQuery.recentProgress¶

Returns an array of the most recent [[StreamingQueryProgress]] updates for this query. The number of progress updates retained for each stream is configured by Spark session configuration spark.sql.streaming.numRecentProgressUpdates.

New in version 2.1.0.

Changed in version 3.5.0: Supports Spark Connect.

Returns

list: List of dict which is the most recent StreamingQueryProgress updates for this query.

Examples

>>> sdf = spark.readStream.format("rate").load()
>>> sq = sdf.writeStream.format('memory').queryName('this_query').start()

Get an array of the most recent query progress updates for this query

>>> sq.recentProgress
[...]

>>> sq.stop()

pyspark.sql.streaming.StreamingQuery.processAllAvailable

pyspark.sql.streaming.StreamingQuery.runId