pyspark.sql.streaming.DataStreamReader.options#

DataStreamReader.options(**options)[source]#

Adds input options for the underlying data source.

New in version 2.0.0.

Changed in version 3.5.0: Supports Spark Connect.

Notes

This API is evolving.

Examples

>>> spark.readStream.options(x="1", y=2)
<...streaming.readwriter.DataStreamReader object ...>

Specify options in a dictionary.

>>> spark.readStream.options(**{"k1": "v1", "k2": "v2"})
<...streaming.readwriter.DataStreamReader object ...>

The example below specifies ‘rowsPerSecond’ and ‘numPartitions’ options to Rate source in order to generate 10 rows with 10 partitions every second.

>>> import time
>>> q = spark.readStream.format("rate").options(
...    rowsPerSecond=10, numPartitions=10
... ).load().writeStream.format("console").start()
>>> time.sleep(3)
>>> q.stop()