class PartitionKeyedAccumulator[T] extends AccumulatorV2[(Int, T), Map[Int, T]]
An AccumulatorV2 that records one value of type T per partition, keyed by partition id with
LAST-WRITE-WINS merge. When the same partition is recorded more than once -- e.g. duplicate
cross-executor computes, or speculative tasks -- the later value replaces the earlier one rather
than aggregating, so each partition contributes exactly once. The key set is the set of recorded
partitions, and callers fold the values (see foldValues) to derive de-duplicated aggregates;
a plain summing accumulator would instead over-count under duplicate computes.
add is expected to be called once per task (e.g. from a task completion listener) with that
partition's value, so a partition is recorded even when it produced nothing. Updates from
failed/interrupted tasks are dropped by the accumulator framework (it is not
countFailedValues), so only complete per-partition values are ever merged.
Backed by a ConcurrentHashMap, whose per-entry atomicity is sufficient here: add and the
putAll in merge are last-write-wins per key, and the reads (value,
accumulatedNumPartitions, foldValues) only require thread-safety and eventual consistency
-- they are weakly consistent during concurrent updates but exact once all updates have been
merged. This avoids any explicit locking (and the nested-lock pattern a two-map merge would
otherwise need).
- T
the per-partition value type. Must be non-null (
ConcurrentHashMapforbids nulls).
- Alphabetic
- By Inheritance
- PartitionKeyedAccumulator
- AccumulatorV2
- Serializable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new PartitionKeyedAccumulator()
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def accumulatedNumPartitions: Long
Number of distinct partitions that have been recorded.
- def add(v: (Int, T)): Unit
Takes the inputs and accumulates.
Takes the inputs and accumulates.
- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- def copy(): PartitionKeyedAccumulator[T]
Creates a new copy of this accumulator.
Creates a new copy of this accumulator.
- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- def copyAndReset(): PartitionKeyedAccumulator[T]
Creates a new copy of this accumulator, which is zero value.
Creates a new copy of this accumulator, which is zero value. i.e. call
isZeroon the copy must return true.- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def excludeFromHeartbeat: Boolean
- Definition Classes
- AccumulatorV2
- def foldValues[A](zero: A)(op: (A, T) => A): A
Folds the per-partition values (each partition counted once) into a single aggregate.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def id: Long
Returns the id of this accumulator, can only be called after registration.
Returns the id of this accumulator, can only be called after registration.
- Definition Classes
- AccumulatorV2
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def isRegistered: Boolean
Returns true if this accumulator has been registered.
Returns true if this accumulator has been registered.
- Definition Classes
- AccumulatorV2
- Note
All accumulators must be registered before use, or it will throw exception.
- def isZero: Boolean
Returns if this accumulator is zero value or not.
Returns if this accumulator is zero value or not. e.g. for a counter accumulator, 0 is zero value; for a list accumulator, Nil is zero value.
- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- def merge(other: AccumulatorV2[(Int, T), Map[Int, T]]): Unit
Merges another same-type accumulator into this one and update its state, i.e.
Merges another same-type accumulator into this one and update its state, i.e. this should be merge-in-place.
- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- final def name: Option[String]
Returns the name of this accumulator, can only be called after registration.
Returns the name of this accumulator, can only be called after registration.
- Definition Classes
- AccumulatorV2
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def reset(): Unit
Resets this accumulator, which is zero value.
Resets this accumulator, which is zero value. i.e. call
isZeromust return true.- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AccumulatorV2 → AnyRef → Any
- def value: Map[Int, T]
Defines the current value of this accumulator
Defines the current value of this accumulator
- Definition Classes
- PartitionKeyedAccumulator → AccumulatorV2
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def withBufferSerialized(): AccumulatorV2[(Int, T), Map[Int, T]]
- Attributes
- protected
- Definition Classes
- AccumulatorV2
- final def writeReplace(): Any
- Attributes
- protected
- Definition Classes
- AccumulatorV2
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)