org.apache.spark.sql.expressions

Class Aggregator<IN,BUF,OUT>

Object

org.apache.spark.sql.expressions.Aggregator<IN,BUF,OUT>

Type Parameters:

IN - The input type for the aggregation.

BUF - The type of the intermediate value of the reduction.

OUT - The type of the final output result.

All Implemented Interfaces:

java.io.Serializable

Direct Known Subclasses:

StringIndexerAggregator

public abstract class Aggregator<IN,BUF,OUT>
extends Object
implements scala.Serializable

A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value.

For example, the following aggregator extracts an int from a specific class and adds them up:

   case class Data(i: Int)

   val customSummer =  new Aggregator[Data, Int, Int] {
     def zero: Int = 0
     def reduce(b: Int, a: Data): Int = b + a.i
     def merge(b1: Int, b2: Int): Int = b1 + b2
     def finish(r: Int): Int = r
     def bufferEncoder: Encoder[Int] = Encoders.scalaInt
     def outputEncoder: Encoder[Int] = Encoders.scalaInt
   }.toColumn()

   val ds: Dataset[Data] = ...
   val aggregated = ds.select(customSummer)
 

Based loosely on Aggregator from Algebird: https://github.com/twitter/algebird

Since:

1.6.0

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Aggregator()

Method Summary

Modifier and Type	Method and Description
abstract Encoder<BUF>	bufferEncoder() Specifies the Encoder for the intermediate value type.
abstract OUT	finish(BUF reduction) Transform the output of the reduction.
abstract BUF	merge(BUF b1, BUF b2) Merge two intermediate values.
abstract Encoder<OUT>	outputEncoder() Specifies the Encoder for the final output value type.
abstract BUF	reduce(BUF b, IN a) Combine two values to produce a new value.
TypedColumn<IN,OUT>	toColumn() Returns this Aggregator as a TypedColumn that can be used in Dataset.
abstract BUF	zero() A zero value for this aggregation.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Aggregator

public Aggregator()

Method Detail

bufferEncoder

public abstract Encoder<BUF> bufferEncoder()

Specifies the Encoder for the intermediate value type.

Returns:

(undocumented)

Since:

2.0.0

finish

public abstract OUT finish(BUF reduction)

Transform the output of the reduction.

Parameters:

reduction - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

merge

public abstract BUF merge(BUF b1,
                          BUF b2)

Merge two intermediate values.

Parameters:

b1 - (undocumented)

b2 - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

outputEncoder

public abstract Encoder<OUT> outputEncoder()

Specifies the Encoder for the final output value type.

Returns:

(undocumented)

Since:

2.0.0

reduce

public abstract BUF reduce(BUF b,
                           IN a)

Combine two values to produce a new value. For performance, the function may modify b and return it instead of constructing new object for b.

Parameters:

b - (undocumented)

a - (undocumented)

Returns:

(undocumented)

Since:

1.6.0

toColumn

public TypedColumn<IN,OUT> toColumn()

Returns this Aggregator as a TypedColumn that can be used in Dataset. operations.

Returns:

(undocumented)

Since:

1.6.0

zero

public abstract BUF zero()

A zero value for this aggregation. Should satisfy the property that any b + zero = b.

Returns:

(undocumented)

Since:

1.6.0