spark

Type Members

class Accumulable[R, T] extends Serializable

A datatype that can be accumulated, i.
trait AccumulableParam[R, T] extends Serializable

Helper object defining how to accumulate values of a particular type.
class Accumulator[T] extends Accumulable[T, T]

A simpler value of Accumulable where the result type being accumulated is the same as the types of elements being merged.
trait AccumulatorParam[T] extends AccumulableParam[T, T]

A simpler version of AccumulableParam where the only datatype you can add in is the same type as the accumulated value.
case class Aggregator[K, V, C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C) extends Product with Serializable

A set of functions used to aggregate data.
abstract class Dependency[T] extends Serializable

Base class for dependencies.
class DoubleRDDFunctions extends Logging with Serializable

Extra functions available on RDDs of Doubles through an implicit conversion.
class HashPartitioner extends Partitioner

A Partitioner that implements hash-based partitioning using Java's Object.hashCode.
class JavaSerializer extends Serializer

A Spark serializer that uses Java's built-in serialization.
trait KryoRegistrator extends AnyRef

Interface implemented by clients to register their classes with Kryo when using Kryo serialization.
class KryoSerializer extends Serializer with Logging

A Spark serializer that uses the Kryo 1.x library.
trait Logging extends AnyRef

Utility trait for classes that want to log data.
abstract class NarrowDependency[T] extends Dependency[T]

Base class for dependencies where each partition of the parent RDD is used by at most one partition of the child RDD.
class OneToOneDependency[T] extends NarrowDependency[T]

Represents a one-to-one dependency between partitions of the parent and child RDDs.
class OrderedRDDFunctions[K, V] extends Logging with Serializable

Extra functions available on RDDs of (key, value) pairs where the key is sortable through an implicit conversion.
class PairRDDFunctions[K, V] extends Logging with HadoopMapReduceUtil with Serializable

Extra functions available on RDDs of (key, value) pairs through an implicit conversion.
trait Partition extends Serializable

A partition of an RDD.
abstract class Partitioner extends Serializable

An object that defines how the elements in a key-value pair RDD are partitioned by key.
abstract class RDD[T] extends Serializable with Logging

A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
class RangeDependency[T] extends NarrowDependency[T]

Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.
class RangePartitioner[K, V] extends Partitioner

A Partitioner that partitions sortable records by range into roughly equal ranges.
class SequenceFileRDDFunctions[K, V] extends Logging with Serializable

Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion.
class SerializableWritable[T <: Writable] extends Serializable
class ShuffleDependency[K, V] extends Dependency[(K, V)]

Represents a dependency on the output of a shuffle stage.
class SparkContext extends Logging

Main entry point for Spark functionality.
class SparkEnv extends AnyRef

Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, Akka actor system, block manager, map output tracker, etc.
class SparkException extends Exception
class SparkFiles extends AnyRef
class TaskContext extends Serializable

Value Members

object Partitioner extends Serializable
object SparkContext extends AnyRef

The SparkContext object contains a number of implicit conversions and parameters for use with various Spark features.
object SparkEnv extends Logging
object SparkFiles extends
package api
package broadcast
package common
package deploy
package executor
package partial
package rdd
package scheduler
package serializer
package storage
package util

spark

package spark

Type Members

class Accumulable[R, T] extends Serializable

trait AccumulableParam[R, T] extends Serializable

class Accumulator[T] extends Accumulable[T, T]

trait AccumulatorParam[T] extends AccumulableParam[T, T]

case class Aggregator[K, V, C](createCombiner: (V) ⇒ C, mergeValue: (C, V) ⇒ C, mergeCombiners: (C, C) ⇒ C) extends Product with Serializable

abstract class Dependency[T] extends Serializable

class DoubleRDDFunctions extends Logging with Serializable

class HashPartitioner extends Partitioner

class JavaSerializer extends Serializer

trait KryoRegistrator extends AnyRef

class KryoSerializer extends Serializer with Logging

trait Logging extends AnyRef

abstract class NarrowDependency[T] extends Dependency[T]

class OneToOneDependency[T] extends NarrowDependency[T]

class OrderedRDDFunctions[K, V] extends Logging with Serializable

class PairRDDFunctions[K, V] extends Logging with HadoopMapReduceUtil with Serializable

trait Partition extends Serializable

abstract class Partitioner extends Serializable

abstract class RDD[T] extends Serializable with Logging

class RangeDependency[T] extends NarrowDependency[T]

class RangePartitioner[K, V] extends Partitioner

class SequenceFileRDDFunctions[K, V] extends Logging with Serializable

class SerializableWritable[T <: Writable] extends Serializable

class ShuffleDependency[K, V] extends Dependency[(K, V)]

class SparkContext extends Logging

class SparkEnv extends AnyRef

class SparkException extends Exception

class SparkFiles extends AnyRef

class TaskContext extends Serializable

Value Members

object Partitioner extends Serializable

object SparkContext extends AnyRef

object SparkEnv extends Logging

object SparkFiles extends

package api

package broadcast

package common

package deploy

package executor

package partial

package rdd

package scheduler

package serializer

package storage

package util