@Evolving public interface PartitionReaderFactory extends java.io.Serializable
If Spark fails to execute any methods in the implementations of this interface or in the returned
PartitionReader (by throwing an exception), corresponding Spark task would fail and
get retried until hitting the maximum retry times.
|Modifier and Type||Method and Description|
Returns a columnar partition reader to read data from the given
Returns a row-based partition reader to read data from the given
Returns true if the given
PartitionReader<org.apache.spark.sql.catalyst.InternalRow> createReader(InputPartition partition)
default PartitionReader<ColumnarBatch> createColumnarReader(InputPartition partition)
default boolean supportColumnarReads(InputPartition partition)
InputPartitionshould be read by Spark in a columnar way. This means, implementations must also implement
createColumnarReader(InputPartition)for the input partitions that this method returns true.
As of Spark 2.4, Spark can only read all input partition in a columnar way, or none of them. Data source can't mix columnar and row-based partitions. This may be relaxed in future versions.