pyspark.sql.streaming.DataStreamReader.options#
- DataStreamReader.options(**options)[source]#
Adds input options for the underlying data source.
New in version 2.0.0.
Changed in version 3.5.0: Supports Spark Connect.
Notes
This API is evolving.
Examples
>>> spark.readStream.options(x="1", y=2) <...streaming.readwriter.DataStreamReader object ...>
Specify options in a dictionary.
>>> spark.readStream.options(**{"k1": "v1", "k2": "v2"}) <...streaming.readwriter.DataStreamReader object ...>
The example below specifies ‘rowsPerSecond’ and ‘numPartitions’ options to Rate source in order to generate 10 rows with 10 partitions every second.
>>> import time >>> q = spark.readStream.format("rate").options( ... rowsPerSecond=10, numPartitions=10 ... ).load().writeStream.format("console").start() >>> time.sleep(3) >>> q.stop()