public class OrcFilters
extends java.lang.Object
SearchArgument
s, which are used for ORC predicate push-down.
Due to limitation of ORC SearchArgument
builder, we had to end up with a pretty weird double-
checking pattern when converting And
/Or
/Not
filters.
An ORC SearchArgument
must be built in one pass using a single builder. For example, you can't
build a = 1
and b = 2
first, and then combine them into a = 1 AND b = 2
. This is quite
different from the cases in Spark SQL or Parquet, where complex filters can be easily built using
existing simpler ones.
The annoying part is that, SearchArgument
builder methods like startAnd()
, startOr()
, and
startNot()
mutate internal state of the builder instance. This forces us to translate all
convertible filters with a single builder instance. However, before actually converting a filter,
we've no idea whether it can be recognized by ORC or not. Thus, when an inconvertible filter is
found, we may already end up with a builder whose internal state is inconsistent.
For example, to convert an And
filter with builder b
, we call b.startAnd()
first, and then
try to convert its children. Say we convert left
child successfully, but find that right
child is inconvertible. Alas, b.startAnd()
call can't be rolled back, and b
is inconsistent
now.
The workaround employed here is that, for And
/Or
/Not
, we first try to convert their
children with brand new builders, and only do the actual conversion with the right builder
instance when the children are proven to be convertible.
P.S.: Hive seems to use SearchArgument
together with ExprNodeGenericFuncDesc
only. Usage of
builder methods mentioned above can only be found in test code, where all tested filters are
known to be convertible.
Constructor and Description |
---|
OrcFilters() |
Modifier and Type | Method and Description |
---|---|
static scala.Option<org.apache.hadoop.hive.ql.io.sarg.SearchArgument> |
createFilter(StructType schema,
Filter[] filters) |
protected static void |
initializeLogIfNecessary(boolean isInterpreter) |
protected static boolean |
isTraceEnabled() |
protected static org.slf4j.Logger |
log() |
protected static void |
logDebug(scala.Function0<java.lang.String> msg) |
protected static void |
logDebug(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logError(scala.Function0<java.lang.String> msg) |
protected static void |
logError(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logInfo(scala.Function0<java.lang.String> msg) |
protected static void |
logInfo(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static java.lang.String |
logName() |
protected static void |
logTrace(scala.Function0<java.lang.String> msg) |
protected static void |
logTrace(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
protected static void |
logWarning(scala.Function0<java.lang.String> msg) |
protected static void |
logWarning(scala.Function0<java.lang.String> msg,
java.lang.Throwable throwable) |
public static scala.Option<org.apache.hadoop.hive.ql.io.sarg.SearchArgument> createFilter(StructType schema, Filter[] filters)
protected static java.lang.String logName()
protected static org.slf4j.Logger log()
protected static void logInfo(scala.Function0<java.lang.String> msg)
protected static void logDebug(scala.Function0<java.lang.String> msg)
protected static void logTrace(scala.Function0<java.lang.String> msg)
protected static void logWarning(scala.Function0<java.lang.String> msg)
protected static void logError(scala.Function0<java.lang.String> msg)
protected static void logInfo(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logDebug(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logTrace(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logWarning(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static void logError(scala.Function0<java.lang.String> msg, java.lang.Throwable throwable)
protected static boolean isTraceEnabled()
protected static void initializeLogIfNecessary(boolean isInterpreter)