pyspark.sql.streaming.DataStreamWriter.outputMode

DataStreamWriter.outputMode(outputMode: str)pyspark.sql.streaming.DataStreamWriter[source]

Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.

New in version 2.0.0.

Options include:

  • append: Only the new rows in the streaming DataFrame/Dataset will be written to

    the sink

  • complete: All the rows in the streaming DataFrame/Dataset will be written to the sink

    every time these are some updates

  • update: only the rows that were updated in the streaming DataFrame/Dataset will be

    written to the sink every time there are some updates. If the query doesn’t contain aggregations, it will be equivalent to append mode.

Notes

This API is evolving.

Examples

>>> writer = sdf.writeStream.outputMode('append')