Interface SupportsAdmissionControl
- All Superinterfaces:
SparkDataStream
- All Known Subinterfaces:
SupportsTriggerAvailableNow
A mix-in interface for
SparkDataStream
streaming sources to signal that they can control
the rate of data ingested into the system. These rate limits can come implicitly from the
contract of triggers, e.g. Trigger.Once() requires that a micro-batch process all data
available to the system at the start of the micro-batch. Alternatively, sources can decide to
limit ingest through data source options.
Through this interface, a MicroBatchStream should be able to return the next offset that it will
process until given a ReadLimit
.
- Since:
- 3.0.0
-
Method Summary
Modifier and TypeMethodDescriptiondefault ReadLimit
Returns the read limits potentially passed to the data source through options when creating the data source.latestOffset
(Offset startOffset, ReadLimit limit) Returns the most recent offset available given a read limit.default Offset
Returns the most recent offset available.Methods inherited from interface org.apache.spark.sql.connector.read.streaming.SparkDataStream
commit, deserializeOffset, initialOffset, stop
-
Method Details
-
getDefaultReadLimit
Returns the read limits potentially passed to the data source through options when creating the data source. -
latestOffset
Returns the most recent offset available given a read limit. The start offset can be used to figure out how much new data should be read given the limit. Users should implement this method instead of latestOffset for a MicroBatchStream or getOffset for Source.When this method is called on a `Source`, the source can return `null` if there is no data to process. In addition, for the very first micro-batch, the `startOffset` will be null as well.
When this method is called on a MicroBatchStream, the `startOffset` will be `initialOffset` for the very first micro-batch. The source can return `null` if there is no data to process.
-
reportLatestOffset
Returns the most recent offset available.The source can return `null`, if there is no data to process or the source does not support to this method.
-