spark.Accumulable

A datatype that can be accumulated, i.e. has an commutative and associative "add" operation, but where the result type, R, may be different from the element type being added, T.

You must define how to add data, and how to merge two of these together. For some datatypes, such as a counter, these might be the same operation. In that case, you can use the simpler Accumulator. They won't always be the same, though -- e.g., imagine you are accumulating a set. You will add items to the set, and you will union two sets together.

R: the full accumulated data (result type)
T: partial data that can be added in

Linear Supertypes

Serializable, AnyRef, Any

Known Subclasses

Accumulator

Instance Constructors

new Accumulable(initialValue: R, param: AccumulableParam[R, T])

initialValue
initial value of accumulator
param
helper object defining how to add elements of type R and T

Value Members

final def !=(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def !=(arg0: Any): Boolean

Definition Classes
Any
final def ##(): Int

Definition Classes
AnyRef → Any
def ++=(term: R): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call +=.
term
the other R that will get merged with this
def +=(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add
final def ==(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def ==(arg0: Any): Boolean

Definition Classes
Any
def add(term: T): Unit

Add more data to this accumulator / accumulable
Add more data to this accumulator / accumulable
term
the data to add
final def asInstanceOf[T0]: T0

Definition Classes
Any
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws()
var deserialized: Boolean
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws()
final def getClass(): java.lang.Class[_]

Definition Classes
AnyRef → Any
def hashCode(): Int

Definition Classes
AnyRef → Any
val id: Long
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
def localValue: R

Get the current value of this accumulator from within a task.
Get the current value of this accumulator from within a task.
This is NOT the global value of the accumulator. To get the global value after a completed operation on the dataset, call value.
The typical use of this method is to directly mutate the local value, eg., to add an element to a Set.
def merge(term: R): Unit

Merge two accumulable objects together
Merge two accumulable objects together
Normally, a user will not want to use this version, but will instead call add.
term
the other R that will get merged with this
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
final def notifyAll(): Unit

Definition Classes
AnyRef
def setValue(newValue: R): Unit

Set the accumulator's value; only allowed on master
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Accumulable → AnyRef → Any
def value: R

Access the accumulator's current value; only allowed on master.
def value_=(newValue: R): Unit

Set the accumulator's value; only allowed on master.
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws()
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws()
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws()
val zero: R

Accumulable

class Accumulable[R, T] extends Serializable

Instance Constructors

new Accumulable(initialValue: R, param: AccumulableParam[R, T])

Value Members

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

def ++=(term: R): Unit

def +=(term: T): Unit

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def add(term: T): Unit

final def asInstanceOf[T0]: T0

def clone(): AnyRef

var deserialized: Boolean

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def finalize(): Unit

final def getClass(): java.lang.Class[_]

def hashCode(): Int

val id: Long

final def isInstanceOf[T0]: Boolean

def localValue: R

def merge(term: R): Unit

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

def setValue(newValue: R): Unit

final def synchronized[T0](arg0: ⇒ T0): T0

def toString(): String

def value: R

def value_=(newValue: R): Unit

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

val zero: R

Inherited from Serializable

Inherited from AnyRef

Inherited from Any