spark.api.python

PythonAccumulatorParam

class PythonAccumulatorParam extends AccumulatorParam[List[Array[Byte]]]

Internal class that acts as an AccumulatorParam for Python accumulators. Inside, it collects a list of pickled strings that we pass to Python through a socket.

Linear Supertypes
AccumulatorParam[List[Array[Byte]]], AccumulableParam[List[Array[Byte]], List[Array[Byte]]], Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. Hide All
  2. Show all
  1. PythonAccumulatorParam
  2. AccumulatorParam
  3. AccumulableParam
  4. Serializable
  5. AnyRef
  6. Any
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PythonAccumulatorParam(serverHost: String, serverPort: Int)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def addAccumulator(t1: List[Array[Byte]], t2: List[Array[Byte]]): List[Array[Byte]]

    Add additional data to the accumulator value.

    Add additional data to the accumulator value. Is allowed to modify and return r for efficiency (to avoid allocating objects).

    returns

    the new value of the accumulator

    Definition Classes
    AccumulatorParamAccumulableParam
  7. def addInPlace(val1: List[Array[Byte]], val2: List[Array[Byte]]): List[Array[Byte]]

    Merge two accumulated values together.

    Merge two accumulated values together. Is allowed to modify and return the first value for efficiency (to avoid allocating objects).

    returns

    both data sets merged together

    Definition Classes
    PythonAccumulatorParamAccumulableParam
  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  9. val bufferSize: Int

  10. def clone(): AnyRef

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  11. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  13. def finalize(): Unit

    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws()
  14. final def getClass(): java.lang.Class[_]

    Definition Classes
    AnyRef → Any
  15. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  16. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  17. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  18. final def notify(): Unit

    Definition Classes
    AnyRef
  19. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  20. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  21. def toString(): String

    Definition Classes
    AnyRef → Any
  22. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  23. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  24. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws()
  25. def zero(value: List[Array[Byte]]): List[Array[Byte]]

    Return the "zero" (identity) value for an accumulator type, given its initial value.

    Return the "zero" (identity) value for an accumulator type, given its initial value. For example, if R was a vector of N dimensions, this would return a vector of N zeroes.

    Definition Classes
    PythonAccumulatorParamAccumulableParam

Inherited from AccumulatorParam[List[Array[Byte]]]

Inherited from AccumulableParam[List[Array[Byte]], List[Array[Byte]]]

Inherited from Serializable

Inherited from AnyRef

Inherited from Any