Package pyspark :: Module accumulators :: Class Accumulator
[frames] | no frames]

Class Accumulator

source code

object --+
         |
        Accumulator

A shared variable that can be accumulated, i.e., has a commutative and associative "add" operation. Worker tasks on a Spark cluster can add values to an Accumulator with the += operator, but only the driver program is allowed to access its value, using value. Updates from the workers get propagated automatically to the driver program.

While SparkContext supports accumulators for primitive data types like int and float, users can also define accumulators for custom types by providing a custom AccumulatorParam object. Refer to the doctest of this module for an example.

Instance Methods
 
__init__(self, aid, value, accum_param)
Create a new Accumulator with a given initial value and AccumulatorParam object
source code
 
__reduce__(self)
Custom serialization; saves the zero value from our AccumulatorParam
source code
 
value(self, value)
Sets the accumulator's value; only usable in driver program
source code
 
__iadd__(self, term)
The += operator; adds a term to this accumulator's value
source code
 
__str__(self)
str(x)
source code
 
__repr__(self)
repr(x)
source code

Inherited from object: __delattr__, __format__, __getattribute__, __hash__, __new__, __reduce_ex__, __setattr__, __sizeof__, __subclasshook__

Properties

Inherited from object: __class__

Method Details

__init__(self, aid, value, accum_param)
(Constructor)

source code 

Create a new Accumulator with a given initial value and AccumulatorParam object

Overrides: object.__init__

__reduce__(self)

source code 

Custom serialization; saves the zero value from our AccumulatorParam

Overrides: object.__reduce__

value(self, value)

source code 

Sets the accumulator's value; only usable in driver program

Decorators:
  • @value.setter

__str__(self)
(Informal representation operator)

source code 

str(x)

Overrides: object.__str__
(inherited documentation)

__repr__(self)
(Representation operator)

source code 

repr(x)

Overrides: object.__repr__
(inherited documentation)