Interface AggregateFunction<S extends Serializable,R>
- Type Parameters:
S
- the JVM type for the aggregation's intermediate state; must beSerializable
R
- the JVM type of result values
- All Superinterfaces:
BoundFunction
,Function
,Serializable
For each input row, Spark will call the update(S, org.apache.spark.sql.catalyst.InternalRow)
method which should evaluate the row
and update the aggregation state. The JVM type of result values produced by
produceResult(S)
must be the type used by Spark's
InternalRow API for the SQL data type
returned by BoundFunction.resultType()
.
Please refer to class documentation of ScalarFunction
for the mapping between
DataType
and the JVM type.
All implementations must support partial aggregation by implementing merge so that Spark can partially aggregate and shuffle intermediate results, instead of shuffling all rows for an aggregate. This reduces the impact of data skew and the amount of data shuffled to produce the result.
Intermediate aggregation state must be Serializable
so that state produced by parallel
tasks can be serialized, shuffled, and then merged to produce a final result.
- Since:
- 3.2.0
-
Method Summary
Modifier and TypeMethodDescriptionMerge two partial aggregation states.Initialize state for an aggregation.produceResult
(S state) Produce the aggregation result based on intermediate state.Update the aggregation state with a new row.Methods inherited from interface org.apache.spark.sql.connector.catalog.functions.BoundFunction
canonicalName, inputTypes, isDeterministic, isResultNullable, resultType
-
Method Details
-
newAggregationState
S newAggregationState()Initialize state for an aggregation.This method is called one or more times for every group of values to initialize intermediate aggregation state. More than one intermediate aggregation state variable may be used when the aggregation is run in parallel tasks.
Implementations that return null must support null state passed into all other methods.
- Returns:
- a state instance or null
-
update
Update the aggregation state with a new row.This is called for each row in a group to update an intermediate aggregation state.
- Parameters:
state
- intermediate aggregation stateinput
- an input row- Returns:
- updated aggregation state
-
merge
Merge two partial aggregation states.This is called to merge intermediate aggregation states that were produced by parallel tasks.
- Parameters:
leftState
- intermediate aggregation staterightState
- intermediate aggregation state- Returns:
- combined aggregation state
-
produceResult
Produce the aggregation result based on intermediate state.- Parameters:
state
- intermediate aggregation state- Returns:
- a result value
-