org.apache.spark.sql.connector.read

Interface PartitionReader<T>

All Superinterfaces:

AutoCloseable, java.io.Closeable

All Known Subinterfaces:

ContinuousPartitionReader<T>

@Evolving
public interface PartitionReader<T>
extends java.io.Closeable

A partition reader returned by PartitionReaderFactory.createReader(InputPartition) or PartitionReaderFactory.createColumnarReader(InputPartition). It's responsible for outputting data for a RDD partition.

Note that, Currently the type `T` can only be InternalRow for normal data sources, or ColumnarBatch for columnar data sources(whose PartitionReaderFactory.supportColumnarReads(InputPartition) returns true).

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
default CustomTaskMetric[]	currentMetricsValues() Returns an array of custom task metrics.
T	get() Return the current record.
boolean	next() Proceed to next record, returns false if there is no more records.

Methods inherited from interface java.io.Closeable

close

Method Detail

next

boolean next()
      throws java.io.IOException

Proceed to next record, returns false if there is no more records.

Throws:

java.io.IOException - if failure happens during disk/network IO like reading files.

get

T get()

Return the current record. This method should return same value until `next` is called.

currentMetricsValues

default CustomTaskMetric[] currentMetricsValues()

Returns an array of custom task metrics. By default it returns empty array. Note that it is not recommended to put heavy logic in this method as it may affect reading performance.