Package org.apache.spark.sql.connector.read.streaming

Interface ContinuousPartitionReader<T>

All Superinterfaces:

AutoCloseable, Closeable, PartitionReader<T>

@Evolving
public interface ContinuousPartitionReader<T>
extends PartitionReader<T>

A variation on PartitionReader for use with continuous streaming processing.

Since:

3.0.0

Method Summary

Modifier and Type	Method	Description
PartitionOffset	getOffset()	Get the offset of the current record, or the start offset if no records have been read.

Methods inherited from interface java.io.Closeable

close

Methods inherited from interface org.apache.spark.sql.connector.read.PartitionReader

currentMetricsValues, get, next

Method Details

getOffset

PartitionOffset getOffset()

Get the offset of the current record, or the start offset if no records have been read.

The execution engine will call this method along with get() to keep track of the current offset. When an epoch ends, the offset of the previous record in each partition will be saved as a restart checkpoint.