UnionRDD

final def !=(arg0: AnyRef): Boolean

final def !=(arg0: Any): Boolean

final def ##(): Int

def ++(other: RDD[T]): RDD[T]

Return the union of this RDD and another one.

final def ==(arg0: AnyRef): Boolean

final def ==(arg0: Any): Boolean

def aggregate[U](zeroValue: U)(seqOp: (U, T) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: ClassManifest[U]): U

Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".

final def asInstanceOf[T0]: T0

def cache(): RDD[T]

Persist this RDD with the default storage level (MEMORY_ONLY).

def cartesian[U](other: RDD[U])(implicit arg0: ClassManifest[U]): RDD[(T, U)]

Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.

def checkpoint(): Unit

Mark this RDD for checkpointing.

def clearDependencies(): Unit

Clears the dependencies of this RDD.

def clone(): AnyRef

def coalesce(numPartitions: Int, shuffle: Boolean = false): RDD[T]

Return a new RDD that is reduced into numPartitions partitions.

def collect[U](f: PartialFunction[T, U])(implicit arg0: ClassManifest[U]): RDD[U]

Return an RDD that contains all matching values by applying f.

def collect(): Array[T]

Return an array that contains all of the elements in this RDD.

def compute(s: Partition, context: TaskContext): Iterator[T]

Implemented by subclasses to compute a given partition.

def context: SparkContext

The SparkContext that this RDD was created on.

def count(): Long

Return the number of elements in the RDD.

def countApprox(timeout: Long, confidence: Double = 0.95): PartialResult[BoundedDouble]

(Experimental) Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.

def countByValue(): Map[T, Long]

Return the count of each unique value in this RDD as a map of (value, count) pairs.

def countByValueApprox(timeout: Long, confidence: Double = 0.95): PartialResult[Map[T, BoundedDouble]]

(Experimental) Approximate version of countByValue().

final def dependencies: Seq[spark.Dependency[_]]

Get the list of dependencies of this RDD, taking into account whether the RDD is checkpointed or not.

def distinct(): RDD[T]

def distinct(numPartitions: Int): RDD[T]

Return a new RDD containing the distinct elements in this RDD.

final def eq(arg0: AnyRef): Boolean

def equals(arg0: Any): Boolean

def filter(f: (T) ⇒ Boolean): RDD[T]

Return a new RDD containing only the elements that satisfy a predicate.

def filterWith[A](constructA: (Int) ⇒ A)(p: (T, A) ⇒ Boolean)(implicit arg0: ClassManifest[A]): RDD[T]

Filters this RDD with p, where p takes an additional parameter of type A.

def finalize(): Unit

def first(): T

Return the first element in this RDD.

def firstParent[U](implicit arg0: ClassManifest[U]): RDD[U]

Returns the first parent RDD

def flatMap[U](f: (T) ⇒ TraversableOnce[U])(implicit arg0: ClassManifest[U]): RDD[U]

Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.

def flatMapWith[A, U](constructA: (Int) ⇒ A, preservesPartitioning: Boolean)(f: (T, A) ⇒ Seq[U])(implicit arg0: ClassManifest[A], arg1: ClassManifest[U]): RDD[U]

FlatMaps f over this RDD, where f takes an additional parameter of type A.

def fold(zeroValue: T)(op: (T, T) ⇒ T): T

Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value".

def foreach(f: (T) ⇒ Unit): Unit

Applies a function f to all elements of this RDD.

def foreachPartition(f: (Iterator[T]) ⇒ Unit): Unit

Applies a function f to each partition of this RDD.

def foreachWith[A](constructA: (Int) ⇒ A)(f: (T, A) ⇒ Unit)(implicit arg0: ClassManifest[A]): Unit

Applies f to each element of this RDD, where f takes an additional parameter of type A.

def getCheckpointFile: Option[String]

Gets the name of the file to which this RDD was checkpointed

final def getClass(): java.lang.Class[_]

def getDependencies: Seq[spark.Dependency[_]]

Implemented by subclasses to return how this RDD depends on parent RDDs.

def getPartitions: Array[Partition]

Implemented by subclasses to return the set of partitions in this RDD.

def getPreferredLocations(s: Partition): Seq[String]

Optionally overridden by subclasses to specify placement preferences.

def getStorageLevel: StorageLevel

Get the RDD's current storage level, or StorageLevel.

def glom(): RDD[Array[T]]

Return an RDD created by coalescing all elements within each partition into an array.

def groupBy[K](f: (T) ⇒ K, p: Partitioner)(implicit arg0: ClassManifest[K]): RDD[(K, Seq[T])]

Return an RDD of grouped items.

def groupBy[K](f: (T) ⇒ K, numPartitions: Int)(implicit arg0: ClassManifest[K]): RDD[(K, Seq[T])]

Return an RDD of grouped elements.

def groupBy[K](f: (T) ⇒ K)(implicit arg0: ClassManifest[K]): RDD[(K, Seq[T])]

Return an RDD of grouped items.

def hashCode(): Int

val id: Int

A unique ID for this RDD (within its SparkContext).

def initLogging(): Unit

def isCheckpointed: Boolean

Return whether this RDD has been checkpointed or not

final def isInstanceOf[T0]: Boolean

final def iterator(split: Partition, context: TaskContext): Iterator[T]

Internal method to this RDD; will read from cache if applicable, or otherwise compute it.

def keyBy[K](f: (T) ⇒ K): RDD[(K, T)]

Creates tuples of the elements in this RDD by applying f.

def log: Logger

def logDebug(msg: ⇒ String, throwable: Throwable): Unit

def logDebug(msg: ⇒ String): Unit

def logError(msg: ⇒ String, throwable: Throwable): Unit

def logError(msg: ⇒ String): Unit

def logInfo(msg: ⇒ String, throwable: Throwable): Unit

def logInfo(msg: ⇒ String): Unit

def logTrace(msg: ⇒ String, throwable: Throwable): Unit

def logTrace(msg: ⇒ String): Unit

def logWarning(msg: ⇒ String, throwable: Throwable): Unit

def logWarning(msg: ⇒ String): Unit

def map[U](f: (T) ⇒ U)(implicit arg0: ClassManifest[U]): RDD[U]

Return a new RDD by applying a function to all elements of this RDD.

def mapPartitions[U](f: (Iterator[T]) ⇒ Iterator[U], preservesPartitioning: Boolean)(implicit arg0: ClassManifest[U]): RDD[U]

Return a new RDD by applying a function to each partition of this RDD.

def mapPartitionsWithIndex[U](f: (Int, Iterator[T]) ⇒ Iterator[U], preservesPartitioning: Boolean)(implicit arg0: ClassManifest[U]): RDD[U]

Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.

def mapWith[A, U](constructA: (Int) ⇒ A, preservesPartitioning: Boolean)(f: (T, A) ⇒ U)(implicit arg0: ClassManifest[A], arg1: ClassManifest[U]): RDD[U]

Maps f over this RDD, where f takes an additional parameter of type A.

var name: String

A friendly name for this RDD

final def ne(arg0: AnyRef): Boolean

final def notify(): Unit

final def notifyAll(): Unit

val partitioner: Option[Partitioner]

Optionally overridden by subclasses to specify how they are partitioned.

final def partitions: Array[Partition]

Get the array of partitions of this RDD, taking into account whether the RDD is checkpointed or not.

def persist(): RDD[T]

Persist this RDD with the default storage level (MEMORY_ONLY).

def persist(newLevel: StorageLevel): RDD[T]

Set this RDD's storage level to persist its values across operations after the first time it is computed.

def pipe(command: Seq[String], env: Map[String, String]): RDD[String]

Return an RDD created by piping elements to a forked external process.

def pipe(command: Seq[String]): RDD[String]

Return an RDD created by piping elements to a forked external process.

def pipe(command: String): RDD[String]

Return an RDD created by piping elements to a forked external process.

final def preferredLocations(split: Partition): Seq[String]

Get the preferred location of a split, taking into account whether the RDD is checkpointed or not.

var rdds: Seq[RDD[T]]

def reduce(f: (T, T) ⇒ T): T

Reduces the elements of this RDD using the specified commutative and associative binary operator.

def sample(withReplacement: Boolean, fraction: Double, seed: Int): RDD[T]

Return a sampled subset of this RDD.

def saveAsObjectFile(path: String): Unit

Save this RDD as a SequenceFile of serialized objects.

def saveAsTextFile(path: String): Unit

Save this RDD as a text file, using string representations of elements.

def setName(_name: String): RDD[T]

Assign a name to this RDD

def subtract(other: RDD[T], p: Partitioner): RDD[T]

Return an RDD with the elements from this that are not in other.

def subtract(other: RDD[T], numPartitions: Int): RDD[T]

Return an RDD with the elements from this that are not in other.

def subtract(other: RDD[T]): RDD[T]

Return an RDD with the elements from this that are not in other.

final def synchronized[T0](arg0: ⇒ T0): T0

def take(num: Int): Array[T]

Take the first num elements of the RDD.

def takeSample(withReplacement: Boolean, num: Int, seed: Int): Array[T]

def toArray(): Array[T]

Return an array that contains all of the elements in this RDD.

def toDebugString: String

A description of this RDD and its recursive dependencies for debugging.

def toString(): String

def union(other: RDD[T]): RDD[T]

Return the union of this RDD and another one.

final def wait(): Unit

final def wait(arg0: Long, arg1: Int): Unit

final def wait(arg0: Long): Unit

def zip[U](other: RDD[U])(implicit arg0: ClassManifest[U]): RDD[(T, U)]

Zips this RDD with another one, returning key-value pairs with the first element in each RDD, second element in each RDD, etc.

def mapPartitionsWithSplit[U](f: (Int, Iterator[T]) ⇒ Iterator[U], preservesPartitioning: Boolean)(implicit arg0: ClassManifest[U]): RDD[U]

Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.