Type Members

1.  class CaseInsensitiveStringMap extends Map[String, String]

   Case-insensitive map of string keys to string values.

   Case-insensitive map of string keys to string values.

   This is used to pass options to v2 implementations to ensure consistent case insensitivity.

   Methods that return keys in this map, like #entrySet() and #keySet(), return keys converted to lower case. This map doesn't allow null key.

   Annotations

   @Experimental()

   Since

   3.0.0

2.  class ExecutionListenerManager extends Logging

   Manager for QueryExecutionListener.

   Manager for QueryExecutionListener. See org.apache.spark.sql.SQLContext.listenerManager.

3.  class MapperRowCounter extends AccumulatorV2[Long, List[(Integer, Long)]]

   An AccumulatorV2 counter for collecting a list of (mapper index, row count).

   An AccumulatorV2 counter for collecting a list of (mapper index, row count).

   Since

   3.4.0

4.  class NumericHistogram extends AnyRef

   A generic, re-usable histogram class that supports partial aggregations.

   A generic, re-usable histogram class that supports partial aggregations. The algorithm is a heuristic adapted from the following paper: Yael Ben-Haim and Elad Tom-Tov, "A streaming parallel decision tree algorithm", J. Machine Learning Research 11 (2010), pp. 849--872. Although there are no approximation guarantees, it appears to work well with adequate data and a large (e.g., 20-80) number of histogram bins.

   Adapted from Hive's NumericHistogram. Can refer to https://github.com/apache/hive/blob/master/ql/src/ java/org/apache/hadoop/hive/ql/udf/generic/NumericHistogram.java

   Differences:

   1. Declaring Coord and it's variables as public types for easy access in the HistogramNumeric class. 2. Add method getNumBins() for serialize NumericHistogram in NumericHistogramSerializer. 3. Add method addBin() for deserialize NumericHistogram in NumericHistogramSerializer. 4. In Hive's code, the method pass a serialized histogram, in Spark, this method pass a deserialized histogram. Here we change the code about merge bins.

   Since

   3.3.0

5.  class PartitionKeyedAccumulator[T] extends AccumulatorV2[(Int, T), Map[Int, T]]

   An AccumulatorV2 that records one value of type T per partition, keyed by partition id with LAST-WRITE-WINS merge.

   An AccumulatorV2 that records one value of type T per partition, keyed by partition id with LAST-WRITE-WINS merge. When the same partition is recorded more than once -- e.g. duplicate cross-executor computes, or speculative tasks -- the later value replaces the earlier one rather than aggregating, so each partition contributes exactly once. The key set is the set of recorded partitions, and callers fold the values (see foldValues) to derive de-duplicated aggregates; a plain summing accumulator would instead over-count under duplicate computes.

   add is expected to be called once per task (e.g. from a task completion listener) with that partition's value, so a partition is recorded even when it produced nothing. Updates from failed/interrupted tasks are dropped by the accumulator framework (it is not countFailedValues), so only complete per-partition values are ever merged.

   Backed by a ConcurrentHashMap, whose per-entry atomicity is sufficient here: add and the putAll in merge are last-write-wins per key, and the reads (value, accumulatedNumPartitions, foldValues) only require thread-safety and eventual consistency -- they are weakly consistent during concurrent updates but exact once all updates have been merged. This avoids any explicit locking (and the nested-lock pattern a two-map merge would otherwise need).

   T

   the per-partition value type. Must be non-null (ConcurrentHashMap forbids nulls).

6.  trait QueryExecutionListener extends AnyRef

   The interface of query execution listener that can be used to analyze execution metrics.

   The interface of query execution listener that can be used to analyze execution metrics.

   Note

   Implementations should guarantee thread-safety as they can be invoked by multiple different threads.

7.  class SQLOpenHashSet[T] extends AnyRef

   Annotations

   @Private()