Package org.apache.spark.sql.connector.read

Interface PartitionReader<T>

All Superinterfaces:
AutoCloseable, Closeable
All Known Subinterfaces:
ContinuousPartitionReader<T>, SupportsRealTimeRead<T>
@Evolving public interface PartitionReader<T> extends Closeable
A partition reader returned by PartitionReaderFactory.createReader(InputPartition) or PartitionReaderFactory.createColumnarReader(InputPartition). It's responsible for outputting data for a RDD partition.

Note that, Currently the type `T` can only be InternalRow for normal data sources, or ColumnarBatch for columnar data sources(whose PartitionReaderFactory.supportColumnarReads(InputPartition) returns true).

Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    default CustomTaskMetric[]
    currentMetricsValues()
    Returns an array of custom task metrics.
    T
    get()
    Return the current record.
    default void
    initMetricsValues(CustomTaskMetric[] metrics)
    Sets the initial value of metrics before fetching any data from the reader.
    boolean
    next()
    Proceed to next record, returns false if there is no more records.

    Methods inherited from interface java.io.Closeable

    close

  • Method Details

    • next

      boolean next() throws IOException
      Proceed to next record, returns false if there is no more records.
      Throws:
      IOException - if failure happens during disk/network IO like reading files.

    • get

      T get()
      Return the current record. This method should return same value until `next` is called.

    • currentMetricsValues

      default CustomTaskMetric[] currentMetricsValues()
      Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.

    • initMetricsValues

      default void initMetricsValues(CustomTaskMetric[] metrics)
      Sets the initial value of metrics before fetching any data from the reader. This is called when multiple PartitionReaders are grouped into one partition in case of KeyGroupedPartitioning and the reader is initialized with the metrics returned by the previous reader that belongs to the same partition. By default, this method does nothing.