Package org.apache.spark.sql.connector.read

Interface SupportsPushDownRequiredColumns

All Superinterfaces:
ScanBuilder
@Evolving public interface SupportsPushDownRequiredColumns extends ScanBuilder
A mix-in interface for ScanBuilder. Data sources can implement this interface to push down required columns to the data source and only read these columns during scan to reduce the size of the data to be read.
Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    pruneColumns(StructType requiredSchema)
    Applies column pruning w.r.t. the given requiredSchema.

    Methods inherited from interface org.apache.spark.sql.connector.read.ScanBuilder

    build

  • Method Details

    • pruneColumns

      void pruneColumns(StructType requiredSchema)
      Applies column pruning w.r.t. the given requiredSchema.

      Implementation should try its best to prune the unnecessary columns or nested fields, but it's also OK to do the pruning partially, e.g., a data source may not be able to prune nested fields, and only prune top-level columns.

      Note that, Scan.readSchema() implementation should take care of the column pruning applied here.