Package org.apache.spark.sql.streaming
Class Trigger
Object
org.apache.spark.sql.streaming.Trigger
Policy used to indicate how often results should be produced by a [[StreamingQuery]].
- Since:
- 2.0.0
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionstatic Trigger
A trigger that processes all available data at the start of the query in one or multiple batches, then terminates the query.static Trigger
Continuous
(long intervalMs) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.static Trigger
Continuous
(long interval, TimeUnit timeUnit) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.static Trigger
Continuous
(String interval) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.static Trigger
Continuous
(scala.concurrent.duration.Duration interval) (Scala-friendly) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.static Trigger
Once()
Deprecated.This is deprecated as of Spark 3.4.0.static Trigger
ProcessingTime
(long intervalMs) A trigger policy that runs a query periodically based on an interval in processing time.static Trigger
ProcessingTime
(long interval, TimeUnit timeUnit) (Java-friendly) A trigger policy that runs a query periodically based on an interval in processing time.static Trigger
ProcessingTime
(String interval) A trigger policy that runs a query periodically based on an interval in processing time.static Trigger
ProcessingTime
(scala.concurrent.duration.Duration interval) (Scala-friendly) A trigger policy that runs a query periodically based on an interval in processing time.
-
Constructor Details
-
Trigger
public Trigger()
-
-
Method Details
-
ProcessingTime
A trigger policy that runs a query periodically based on an interval in processing time. If `interval` is 0, the query will run as fast as possible.- Since:
- 2.2.0
-
ProcessingTime
(Java-friendly) A trigger policy that runs a query periodically based on an interval in processing time. If `interval` is 0, the query will run as fast as possible. {{{ import java.util.concurrent.TimeUnit df.writeStream().trigger(Trigger.ProcessingTime(10, TimeUnit.SECONDS)) }}}- Since:
- 2.2.0
-
ProcessingTime
(Scala-friendly) A trigger policy that runs a query periodically based on an interval in processing time. If `duration` is 0, the query will run as fast as possible. {{{ import scala.concurrent.duration._ df.writeStream.trigger(Trigger.ProcessingTime(10.seconds)) }}}- Since:
- 2.2.0
-
ProcessingTime
A trigger policy that runs a query periodically based on an interval in processing time. If `interval` is effectively 0, the query will run as fast as possible. {{{ df.writeStream.trigger(Trigger.ProcessingTime("10 seconds")) }}}- Since:
- 2.2.0
-
Once
Deprecated.This is deprecated as of Spark 3.4.0. UseAvailableNow()
to leverage better guarantee of processing, fine-grained scale of batches, and better gradual processing of watermark advancement including no-data batch. See the NOTES inAvailableNow()
for details.A trigger that processes all available data in a single batch then terminates the query.- Since:
- 2.2.0
-
AvailableNow
A trigger that processes all available data at the start of the query in one or multiple batches, then terminates the query. Users are encouraged to set the source options to control the size of the batch as similar as controlling the size of the batch inProcessingTime(long)
trigger. NOTES: - This trigger provides a strong guarantee of processing: regardless of how many batches were left over in previous run, it ensures all available data at the time of execution gets processed before termination. All uncommitted batches will be processed first. - Watermark gets advanced per each batch, and no-data batch gets executed before termination if the last batch advances the watermark. This helps to maintain smaller and predictable state size and smaller latency on the output of stateful operators.- Since:
- 3.3.0
-
Continuous
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.- Since:
- 2.3.0
-
Continuous
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval. {{{ import java.util.concurrent.TimeUnit df.writeStream.trigger(Trigger.Continuous(10, TimeUnit.SECONDS)) }}}- Since:
- 2.3.0
-
Continuous
(Scala-friendly) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval. {{{ import scala.concurrent.duration._ df.writeStream.trigger(Trigger.Continuous(10.seconds)) }}}- Since:
- 2.3.0
-
Continuous
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval. {{{ df.writeStream.trigger(Trigger.Continuous("10 seconds")) }}}- Since:
- 2.3.0
-