Package org.apache.spark.sql.connector.read.streaming

Interface SupportsRealTimeMode

@Evolving public interface SupportsRealTimeMode
A MicroBatchStream for streaming queries with real time mode.

  • Method Details

    • planInputPartitions

      InputPartition[] planInputPartitions(Offset start)
      Returns a list of input partitions given the start offset. Each InputPartition represents a data split that can be processed by one Spark task. The number of input partitions returned here is the same as the number of RDD partitions this scan outputs.

    • mergeOffsets

      Offset mergeOffsets(PartitionOffset[] offsets)
      Merge partitioned offsets coming from SupportsRealTimeMode instances for each partition to a single global offset.

    • prepareForRealTimeMode

      default void prepareForRealTimeMode()
      Called during logical planning to inform the source if it's in real time mode