org.apache.spark.sql.connector.read.streaming

Interface AcceptsLatestSeenOffset

All Superinterfaces:

SparkDataStream

public interface AcceptsLatestSeenOffset
extends SparkDataStream

Indicates that the source accepts the latest seen offset, which requires streaming execution to provide the latest seen offset when restarting the streaming query from checkpoint. Note that this interface aims to only support DSv2 streaming sources. Spark may throw error if the interface is implemented along with DSv1 streaming sources. The callback method will be called once per run.

Since:

3.3.0

Method Summary

Modifier and Type	Method and Description
void	setLatestSeenOffset(Offset offset) Callback method to receive the latest seen offset information from streaming execution.

Methods inherited from interface org.apache.spark.sql.connector.read.streaming.SparkDataStream

commit, deserializeOffset, initialOffset, stop

Method Detail

setLatestSeenOffset

void setLatestSeenOffset(Offset offset)

Callback method to receive the latest seen offset information from streaming execution. The method will be called only when the streaming query is restarted from checkpoint.

Parameters:

offset - The offset which was latest seen in the previous run.