pyspark.sql.streaming.DataStreamWriter.option¶
-
DataStreamWriter.option(key: str, value: OptionalPrimitiveType) → DataStreamWriter[source]¶ Adds an output option for the underlying data source.
New in version 2.0.0.
Changed in version 3.5.0: Supports Spark Connect.
Notes
This API is evolving.
Examples
>>> df = spark.readStream.format("rate").load() >>> df.writeStream.option("x", 1) <...streaming.readwriter.DataStreamWriter object ...>
The example below specifies ‘numRows’ option to Console source in order to print 3 rows for every batch.
>>> import time >>> q = spark.readStream.format( ... "rate").option("rowsPerSecond", 10).load().writeStream.format( ... "console").option("numRows", 3).start() >>> time.sleep(3) >>> q.stop()