|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
Object org.apache.spark.api.java.JavaPairRDD<K,V>
public class JavaPairRDD<K,V>
Constructor Summary | |
---|---|
JavaPairRDD(RDD<scala.Tuple2<K,V>> rdd,
scala.reflect.ClassTag<K> kClassTag,
scala.reflect.ClassTag<V> vClassTag)
|
Method Summary | |||||||||
---|---|---|---|---|---|---|---|---|---|
|
aggregateByKey(U zeroValue,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value". |
||||||||
|
aggregateByKey(U zeroValue,
int numPartitions,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value". |
||||||||
|
aggregateByKey(U zeroValue,
Partitioner partitioner,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value". |
||||||||
JavaPairRDD<K,V> |
cache()
Persist this RDD with the default storage level (`MEMORY_ONLY`). |
||||||||
scala.reflect.ClassTag<scala.Tuple2<K,V>> |
classTag()
|
||||||||
JavaPairRDD<K,V> |
coalesce(int numPartitions)
Return a new RDD that is reduced into numPartitions partitions. |
||||||||
JavaPairRDD<K,V> |
coalesce(int numPartitions,
boolean shuffle)
Return a new RDD that is reduced into numPartitions partitions. |
||||||||
|
cogroup(JavaPairRDD<K,W> other)
For each key k in this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
||||||||
|
cogroup(JavaPairRDD<K,W> other,
int numPartitions)
For each key k in this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
||||||||
|
cogroup(JavaPairRDD<K,W> other,
Partitioner partitioner)
For each key k in this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2)
For each key k in this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
int numPartitions)
For each key k in this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3)
For each key k in this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3,
int numPartitions)
For each key k in this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3,
Partitioner partitioner)
For each key k in this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
||||||||
|
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
Partitioner partitioner)
For each key k in this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
||||||||
java.util.Map<K,V> |
collectAsMap()
Return the key-value pairs in this RDD to the master as a Map. |
||||||||
|
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners)
Simplified version of combineByKey that hash-partitions the resulting RDD using the existing partitioner/parallelism level and using map-side aggregation. |
||||||||
|
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
int numPartitions)
Simplified version of combineByKey that hash-partitions the output RDD and uses map-side aggregation. |
||||||||
|
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
Partitioner partitioner)
Generic function to combine the elements for each key using a custom set of aggregation functions. |
||||||||
|
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
Partitioner partitioner,
boolean mapSideCombine,
Serializer serializer)
Generic function to combine the elements for each key using a custom set of aggregation functions. |
||||||||
JavaPairRDD<K,Object> |
countApproxDistinctByKey(double relativeSD)
Return approximate number of distinct values for each key in this RDD. |
||||||||
JavaPairRDD<K,Object> |
countApproxDistinctByKey(double relativeSD,
int numPartitions)
Return approximate number of distinct values for each key in this RDD. |
||||||||
JavaPairRDD<K,Object> |
countApproxDistinctByKey(double relativeSD,
Partitioner partitioner)
Return approximate number of distinct values for each key in this RDD. |
||||||||
java.util.Map<K,Object> |
countByKey()
Count the number of elements for each key, and return the result to the master as a Map. |
||||||||
PartialResult<java.util.Map<K,BoundedDouble>> |
countByKeyApprox(long timeout)
:: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout. |
||||||||
PartialResult<java.util.Map<K,BoundedDouble>> |
countByKeyApprox(long timeout,
double confidence)
:: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout. |
||||||||
JavaPairRDD<K,V> |
distinct()
Return a new RDD containing the distinct elements in this RDD. |
||||||||
JavaPairRDD<K,V> |
distinct(int numPartitions)
Return a new RDD containing the distinct elements in this RDD. |
||||||||
JavaPairRDD<K,V> |
filter(Function<scala.Tuple2<K,V>,Boolean> f)
Return a new RDD containing only the elements that satisfy a predicate. |
||||||||
scala.Tuple2<K,V> |
first()
Return the first element in this RDD. |
||||||||
|
flatMapValues(Function<V,Iterable<U>> f)
Pass each value in the key-value pair RDD through a flatMap function without changing the keys; this also retains the original RDD's partitioning. |
||||||||
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.). |
||||||||
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
int numPartitions,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.). |
||||||||
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
Partitioner partitioner,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.). |
||||||||
static
|
fromJavaRDD(JavaRDD<scala.Tuple2<K,V>> rdd)
Convert a JavaRDD of key-value pairs to JavaPairRDD. |
||||||||
static
|
fromRDD(RDD<scala.Tuple2<K,V>> rdd,
scala.reflect.ClassTag<K> evidence$5,
scala.reflect.ClassTag<V> evidence$6)
|
||||||||
|
fullOuterJoin(JavaPairRDD<K,W> other)
Perform a full outer join of this and other . |
||||||||
|
fullOuterJoin(JavaPairRDD<K,W> other,
int numPartitions)
Perform a full outer join of this and other . |
||||||||
|
fullOuterJoin(JavaPairRDD<K,W> other,
Partitioner partitioner)
Perform a full outer join of this and other . |
Methods inherited from class Object |
---|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Methods inherited from interface org.apache.spark.api.java.JavaRDDLike |
---|
aggregate, cartesian, checkpoint, collect, collectAsync, collectPartitions, context, count, countApprox, countApprox, countApproxDistinct, countAsync, countByValue, countByValueApprox, countByValueApprox, flatMap, flatMapToDouble, flatMapToPair, fold, foreach, foreachAsync, foreachPartition, foreachPartitionAsync, getCheckpointFile, getStorageLevel, glom, groupBy, groupBy, id, isCheckpointed, isEmpty, iterator, keyBy, map, mapPartitions, mapPartitions, mapPartitionsToDouble, mapPartitionsToDouble, mapPartitionsToPair, mapPartitionsToPair, mapPartitionsWithIndex, mapToDouble, mapToPair, max, min, name, partitions, pipe, pipe, pipe, reduce, saveAsObjectFile, saveAsTextFile, saveAsTextFile, splits, take, takeAsync, takeOrdered, takeOrdered, takeSample, takeSample, toArray, toDebugString, toLocalIterator, top, top, treeAggregate, treeAggregate, treeReduce, treeReduce, zip, zipPartitions, zipWithIndex, zipWithUniqueId |
Constructor Detail |
---|
public JavaPairRDD(RDD<scala.Tuple2<K,V>> rdd, scala.reflect.ClassTag<K> kClassTag, scala.reflect.ClassTag<V> vClassTag)
Method Detail |
---|
public static <K,V> JavaPairRDD<K,V> fromRDD(RDD<scala.Tuple2<K,V>> rdd, scala.reflect.ClassTag<K> evidence$5, scala.reflect.ClassTag<V> evidence$6)
public static <K,V> RDD<scala.Tuple2<K,V>> toRDD(JavaPairRDD<K,V> rdd)
public static <K,V> JavaPairRDD<K,V> fromJavaRDD(JavaRDD<scala.Tuple2<K,V>> rdd)
public RDD<scala.Tuple2<K,V>> rdd()
public scala.reflect.ClassTag<K> kClassTag()
public scala.reflect.ClassTag<V> vClassTag()
public JavaPairRDD<K,V> wrapRDD(RDD<scala.Tuple2<K,V>> rdd)
public scala.reflect.ClassTag<scala.Tuple2<K,V>> classTag()
public JavaPairRDD<K,V> cache()
public JavaPairRDD<K,V> persist(StorageLevel newLevel)
newLevel
- (undocumented)
public JavaPairRDD<K,V> unpersist()
public JavaPairRDD<K,V> unpersist(boolean blocking)
blocking
- Whether to block until all blocks are deleted.
public JavaPairRDD<K,V> distinct()
public JavaPairRDD<K,V> distinct(int numPartitions)
numPartitions
- (undocumented)
public JavaPairRDD<K,V> filter(Function<scala.Tuple2<K,V>,Boolean> f)
f
- (undocumented)
public JavaPairRDD<K,V> coalesce(int numPartitions)
numPartitions
partitions.
numPartitions
- (undocumented)
public JavaPairRDD<K,V> coalesce(int numPartitions, boolean shuffle)
numPartitions
partitions.
numPartitions
- (undocumented)shuffle
- (undocumented)
public JavaPairRDD<K,V> repartition(int numPartitions)
Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.
If you are decreasing the number of partitions in this RDD, consider using coalesce
,
which can avoid performing a shuffle.
numPartitions
- (undocumented)
public JavaPairRDD<K,V> sample(boolean withReplacement, double fraction)
withReplacement
- (undocumented)fraction
- (undocumented)
public JavaPairRDD<K,V> sample(boolean withReplacement, double fraction, long seed)
withReplacement
- (undocumented)fraction
- (undocumented)seed
- (undocumented)
public JavaPairRDD<K,V> sampleByKey(boolean withReplacement, java.util.Map<K,Object> fractions, long seed)
Create a sample of this RDD using variable sampling rates for different keys as specified by
fractions
, a key to sampling rate map, via simple random sampling with one pass over the
RDD, to produce a sample of size that's approximately equal to the sum of
math.ceil(numItems * samplingRate) over all key values.
withReplacement
- (undocumented)fractions
- (undocumented)seed
- (undocumented)
public JavaPairRDD<K,V> sampleByKey(boolean withReplacement, java.util.Map<K,Object> fractions)
Create a sample of this RDD using variable sampling rates for different keys as specified by
fractions
, a key to sampling rate map, via simple random sampling with one pass over the
RDD, to produce a sample of size that's approximately equal to the sum of
math.ceil(numItems * samplingRate) over all key values.
Use Utils.random.nextLong as the default seed for the random number generator.
withReplacement
- (undocumented)fractions
- (undocumented)
public JavaPairRDD<K,V> sampleByKeyExact(boolean withReplacement, java.util.Map<K,Object> fractions, long seed)
public JavaPairRDD<K,V> sampleByKeyExact(boolean withReplacement, java.util.Map<K,Object> fractions)
public JavaPairRDD<K,V> union(JavaPairRDD<K,V> other)
.distinct()
to eliminate them).
other
- (undocumented)
public JavaPairRDD<K,V> intersection(JavaPairRDD<K,V> other)
Note that this method performs a shuffle internally.
other
- (undocumented)
public scala.Tuple2<K,V> first()
JavaRDDLike
public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, Partitioner partitioner, boolean mapSideCombine, Serializer serializer)
- createCombiner
, which turns a V into a C (e.g., creates a one-element list)
- mergeValue
, to merge a V into a C (e.g., adds it to the end of a list)
- mergeCombiners
, to combine two C's into a single one.
In addition, users can control the partitioning of the output RDD, the serializer that is use for the shuffle, and whether to perform map-side aggregation (if a mapper can produce multiple items with the same key).
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)partitioner
- (undocumented)mapSideCombine
- (undocumented)serializer
- (undocumented)
public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, Partitioner partitioner)
- createCombiner
, which turns a V into a C (e.g., creates a one-element list)
- mergeValue
, to merge a V into a C (e.g., adds it to the end of a list)
- mergeCombiners
, to combine two C's into a single one.
In addition, users can control the partitioning of the output RDD. This method automatically uses map-side aggregation in shuffling the RDD.
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)partitioner
- (undocumented)
public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, int numPartitions)
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)numPartitions
- (undocumented)
public JavaPairRDD<K,V> reduceByKey(Partitioner partitioner, Function2<V,V,V> func)
partitioner
- (undocumented)func
- (undocumented)
public java.util.Map<K,V> reduceByKeyLocally(Function2<V,V,V> func)
func
- (undocumented)
public java.util.Map<K,Object> countByKey()
public PartialResult<java.util.Map<K,BoundedDouble>> countByKeyApprox(long timeout)
timeout
- (undocumented)
public PartialResult<java.util.Map<K,BoundedDouble>> countByKeyApprox(long timeout, double confidence)
timeout
- (undocumented)confidence
- (undocumented)
public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, Partitioner partitioner, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)partitioner
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)
public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, int numPartitions, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)numPartitions
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)
public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)
public JavaPairRDD<K,V> foldByKey(V zeroValue, Partitioner partitioner, Function2<V,V,V> func)
zeroValue
- (undocumented)partitioner
- (undocumented)func
- (undocumented)
public JavaPairRDD<K,V> foldByKey(V zeroValue, int numPartitions, Function2<V,V,V> func)
zeroValue
- (undocumented)numPartitions
- (undocumented)func
- (undocumented)
public JavaPairRDD<K,V> foldByKey(V zeroValue, Function2<V,V,V> func)
zeroValue
- (undocumented)func
- (undocumented)
public JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)
func
- (undocumented)numPartitions
- (undocumented)
public JavaPairRDD<K,Iterable<V>> groupByKey(Partitioner partitioner)
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
partitioner
- (undocumented)
public JavaPairRDD<K,Iterable<V>> groupByKey(int numPartitions)
numPartitions
partitions.
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
numPartitions
- (undocumented)
public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other)
this
that are not in other
.
Uses this
partitioner/partition size, because even if other
is huge, the resulting
RDD will be <= us.
other
- (undocumented)
public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other, int numPartitions)
this
that are not in other
.
other
- (undocumented)numPartitions
- (undocumented)
public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other, Partitioner p)
this
that are not in other
.
other
- (undocumented)p
- (undocumented)
public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other)
this
whose keys are not in other
.
Uses this
partitioner/partition size, because even if other
is huge, the resulting
RDD will be <= us.
other
- (undocumented)
public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other, int numPartitions)
public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other, Partitioner p)
public JavaPairRDD<K,V> partitionBy(Partitioner partitioner)
partitioner
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other, Partitioner partitioner)
other
- (undocumented)partitioner
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<V,com.google.common.base.Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Uses the given Partitioner to
partition the output RDD.
other
- (undocumented)partitioner
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Uses the given Partitioner to
partition the output RDD.
other
- (undocumented)partitioner
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Uses the given Partitioner to partition the output RDD.
other
- (undocumented)partitioner
- (undocumented)
public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners)
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)
public JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func)
func
- (undocumented)
public JavaPairRDD<K,Iterable<V>> groupByKey()
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)
this
and other
. Each
pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this
and
(k, v2) is in other
. Performs a hash join across the cluster.
other
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. Each
pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this
and
(k, v2) is in other
. Performs a hash join across the cluster.
other
- (undocumented)numPartitions
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<V,com.google.common.base.Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Hash-partitions the output
using the existing partitioner/parallelism level.
other
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<V,com.google.common.base.Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Hash-partitions the output
into numPartitions
partitions.
other
- (undocumented)numPartitions
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Hash-partitions the resulting
RDD using the existing partitioner/parallelism level.
other
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Hash-partitions the resulting
RDD into the given number of partitions.
other
- (undocumented)numPartitions
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Hash-partitions the resulting RDD using the existing partitioner/
parallelism level.
other
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<com.google.common.base.Optional<V>,com.google.common.base.Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Hash-partitions the resulting RDD into the given number of partitions.
other
- (undocumented)numPartitions
- (undocumented)
public java.util.Map<K,V> collectAsMap()
public <U> JavaPairRDD<K,U> mapValues(Function<V,U> f)
f
- (undocumented)
public <U> JavaPairRDD<K,U> flatMapValues(Function<V,Iterable<U>> f)
f
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, Partitioner partitioner)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.
other
- (undocumented)partitioner
- (undocumented)
public <W1,W2> JavaPairRDD<K,scala.Tuple3<Iterable<V>,Iterable<W1>,Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, Partitioner partitioner)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.
other1
- (undocumented)other2
- (undocumented)partitioner
- (undocumented)
public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<Iterable<V>,Iterable<W1>,Iterable<W2>,Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3, Partitioner partitioner)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.
other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)partitioner
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.
other
- (undocumented)
public <W1,W2> JavaPairRDD<K,scala.Tuple3<Iterable<V>,Iterable<W1>,Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.
other1
- (undocumented)other2
- (undocumented)
public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<Iterable<V>,Iterable<W1>,Iterable<W2>,Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.
other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.
other
- (undocumented)numPartitions
- (undocumented)
public <W1,W2> JavaPairRDD<K,scala.Tuple3<Iterable<V>,Iterable<W1>,Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, int numPartitions)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.
other1
- (undocumented)other2
- (undocumented)numPartitions
- (undocumented)
public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<Iterable<V>,Iterable<W1>,Iterable<W2>,Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3, int numPartitions)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.
other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)numPartitions
- (undocumented)
public <W> JavaPairRDD<K,scala.Tuple2<Iterable<V>,Iterable<W>>> groupWith(JavaPairRDD<K,W> other)
public <W1,W2> JavaPairRDD<K,scala.Tuple3<Iterable<V>,Iterable<W1>,Iterable<W2>>> groupWith(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2)
public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<Iterable<V>,Iterable<W1>,Iterable<W2>,Iterable<W3>>> groupWith(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3)
public java.util.List<V> lookup(K key)
key
. This operation is done efficiently if the
RDD has a known partitioner by only searching the partition that the key maps to.
key
- (undocumented)
public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(String path, Class<?> keyClass, Class<?> valueClass, Class<F> outputFormatClass, org.apache.hadoop.mapred.JobConf conf)
public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(String path, Class<?> keyClass, Class<?> valueClass, Class<F> outputFormatClass)
public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(String path, Class<?> keyClass, Class<?> valueClass, Class<F> outputFormatClass, Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)
public <F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> void saveAsNewAPIHadoopFile(String path, Class<?> keyClass, Class<?> valueClass, Class<F> outputFormatClass, org.apache.hadoop.conf.Configuration conf)
public void saveAsNewAPIHadoopDataset(org.apache.hadoop.conf.Configuration conf)
conf
- (undocumented)public <F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> void saveAsNewAPIHadoopFile(String path, Class<?> keyClass, Class<?> valueClass, Class<F> outputFormatClass)
public void saveAsHadoopDataset(org.apache.hadoop.mapred.JobConf conf)
conf
- (undocumented)public JavaPairRDD<K,V> repartitionAndSortWithinPartitions(Partitioner partitioner)
This is more efficient than calling repartition
and then sorting within each partition
because it can push the sorting down into the shuffle machinery.
partitioner
- (undocumented)
public JavaPairRDD<K,V> repartitionAndSortWithinPartitions(Partitioner partitioner, java.util.Comparator<K> comp)
This is more efficient than calling repartition
and then sorting within each partition
because it can push the sorting down into the shuffle machinery.
partitioner
- (undocumented)comp
- (undocumented)
public JavaPairRDD<K,V> sortByKey()
collect
or save
on the resulting RDD will return or output an
ordered list of records (in the save
case, they will be written to multiple part-X
files
in the filesystem, in order of the keys).
public JavaPairRDD<K,V> sortByKey(boolean ascending)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).
ascending
- (undocumented)
public JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).
ascending
- (undocumented)numPartitions
- (undocumented)
public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).
comp
- (undocumented)
public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp, boolean ascending)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).
comp
- (undocumented)ascending
- (undocumented)
public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp, boolean ascending, int numPartitions)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).
comp
- (undocumented)ascending
- (undocumented)numPartitions
- (undocumented)
public JavaRDD<K> keys()
public JavaRDD<V> values()
public JavaPairRDD<K,Object> countApproxDistinctByKey(double relativeSD, Partitioner partitioner)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.partitioner
- partitioner of the resulting RDD.
public JavaPairRDD<K,Object> countApproxDistinctByKey(double relativeSD, int numPartitions)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.numPartitions
- number of partitions of the resulting RDD.
public JavaPairRDD<K,Object> countApproxDistinctByKey(double relativeSD)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.
public JavaPairRDD<K,V> setName(String name)
|
|||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | ||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |