public class ParquetFilters
extends Object
Constructor and Description |
---|
ParquetFilters() |
Modifier and Type | Method and Description |
---|---|
static scala.Option<org.apache.spark.sql.parquet.CatalystFilter> |
createFilter(org.apache.spark.sql.catalyst.expressions.Expression expression) |
static parquet.filter.UnboundRecordFilter |
createRecordFilter(scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> filterExpressions) |
static scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> |
deserializeFilterExpressions(org.apache.hadoop.conf.Configuration conf)
Note: Inside the Hadoop API we only have access to
Configuration , not to
SparkContext , so we cannot use broadcasts to convey
the actual filter predicate. |
static scala.Option<org.apache.spark.sql.parquet.CatalystFilter> |
findExpression(org.apache.spark.sql.parquet.CatalystFilter filter,
org.apache.spark.sql.catalyst.expressions.Expression expression)
Try to find the given expression in the tree of filters in order to
determine whether it is safe to remove it from the higher level filters.
|
static String |
PARQUET_FILTER_DATA() |
static String |
PARQUET_FILTER_PUSHDOWN_ENABLED() |
static void |
serializeFilterExpressions(scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> filters,
org.apache.hadoop.conf.Configuration conf)
Note: Inside the Hadoop API we only have access to
Configuration , not to
SparkContext , so we cannot use broadcasts to convey
the actual filter predicate. |
public static String PARQUET_FILTER_DATA()
public static String PARQUET_FILTER_PUSHDOWN_ENABLED()
public static parquet.filter.UnboundRecordFilter createRecordFilter(scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> filterExpressions)
public static scala.Option<org.apache.spark.sql.parquet.CatalystFilter> createFilter(org.apache.spark.sql.catalyst.expressions.Expression expression)
public static void serializeFilterExpressions(scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> filters, org.apache.hadoop.conf.Configuration conf)
Configuration
, not to
SparkContext
, so we cannot use broadcasts to convey
the actual filter predicate.public static scala.collection.Seq<org.apache.spark.sql.catalyst.expressions.Expression> deserializeFilterExpressions(org.apache.hadoop.conf.Configuration conf)
Configuration
, not to
SparkContext
, so we cannot use broadcasts to convey
the actual filter predicate.public static scala.Option<org.apache.spark.sql.parquet.CatalystFilter> findExpression(org.apache.spark.sql.parquet.CatalystFilter filter, org.apache.spark.sql.catalyst.expressions.Expression expression)
filter
- The CatalystFilter
to expand
and searchexpression
- The expression to look forCatalystFilter
that
contains the expression.