@InterfaceStability.Evolving public interface MicroBatchReader extends DataSourceReader, org.apache.spark.sql.execution.streaming.BaseStreamingSource
DataSourceReader
. Data source readers can implement this
interface to indicate they allow micro-batch streaming reads.
Note: This class currently extends BaseStreamingSource
to maintain compatibility with
DataSource V1 APIs. This extension will be removed once we get rid of V1 completely.Modifier and Type | Method and Description |
---|---|
void |
commit(Offset end)
Informs the source that Spark has completed processing all data for offsets less than or
equal to `end` and will only request offsets greater than `end` in the future.
|
Offset |
deserializeOffset(String json)
Deserialize a JSON string into an Offset of the implementation-defined offset type.
|
Offset |
getEndOffset()
Return the specified (if explicitly set through setOffsetRange) or inferred end offset
for this reader.
|
Offset |
getStartOffset()
Returns the specified (if explicitly set through setOffsetRange) or inferred start offset
for this reader.
|
void |
setOffsetRange(java.util.Optional<Offset> start,
java.util.Optional<Offset> end)
Set the desired offset range for input partitions created from this reader.
|
planInputPartitions, readSchema
void setOffsetRange(java.util.Optional<Offset> start, java.util.Optional<Offset> end)
start
- The initial offset to scan from. If not specified, scan from an
implementation-specified start point, such as the earliest available record.end
- The last offset to include in the scan. If not specified, scan up to an
implementation-defined endpoint, such as the last available offset
or the start offset plus a target batch size.Offset getStartOffset()
IllegalStateException
- if setOffsetRange has not been calledOffset getEndOffset()
IllegalStateException
- if setOffsetRange has not been calledOffset deserializeOffset(String json)
IllegalArgumentException
- if the JSON does not encode a valid offset for this readervoid commit(Offset end)