public class JavaPairRDD<K,V>
extends java.lang.Object
Constructor and Description |
---|
JavaPairRDD(RDD<scala.Tuple2<K,V>> rdd,
scala.reflect.ClassTag<K> kClassTag,
scala.reflect.ClassTag<V> vClassTag) |
Modifier and Type | Method and Description |
---|---|
static <U> U |
aggregate(U zeroValue,
Function2<U,T,U> seqOp,
Function2<U,U,U> combOp) |
<U> JavaPairRDD<K,U> |
aggregateByKey(U zeroValue,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value".
|
<U> JavaPairRDD<K,U> |
aggregateByKey(U zeroValue,
int numPartitions,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value".
|
<U> JavaPairRDD<K,U> |
aggregateByKey(U zeroValue,
Partitioner partitioner,
Function2<U,V,U> seqFunc,
Function2<U,U,U> combFunc)
Aggregate the values of each key, using given combine functions and a neutral "zero value".
|
JavaPairRDD<K,V> |
cache()
Persist this RDD with the default storage level (`MEMORY_ONLY`).
|
static <U> JavaPairRDD<T,U> |
cartesian(JavaRDDLike<U,?> other) |
static void |
checkpoint() |
scala.reflect.ClassTag<scala.Tuple2<K,V>> |
classTag() |
JavaPairRDD<K,V> |
coalesce(int numPartitions)
Return a new RDD that is reduced into
numPartitions partitions. |
JavaPairRDD<K,V> |
coalesce(int numPartitions,
boolean shuffle)
Return a new RDD that is reduced into
numPartitions partitions. |
<W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> |
cogroup(JavaPairRDD<K,W> other)
For each key k in
this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
<W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> |
cogroup(JavaPairRDD<K,W> other,
int numPartitions)
For each key k in
this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
<W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> |
cogroup(JavaPairRDD<K,W> other,
Partitioner partitioner)
For each key k in
this or other , return a resulting RDD that contains a tuple with the
list of values for that key in this as well as other . |
<W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2)
For each key k in
this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
<W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
int numPartitions)
For each key k in
this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
<W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3)
For each key k in
this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
<W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3,
int numPartitions)
For each key k in
this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
<W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3,
Partitioner partitioner)
For each key k in
this or other1 or other2 or other3 ,
return a resulting RDD that contains a tuple with the list of values
for that key in this , other1 , other2 and other3 . |
<W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> |
cogroup(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
Partitioner partitioner)
For each key k in
this or other1 or other2 , return a resulting RDD that contains a
tuple with the list of values for that key in this , other1 and other2 . |
static java.util.List<T> |
collect() |
java.util.Map<K,V> |
collectAsMap()
Return the key-value pairs in this RDD to the master as a Map.
|
static JavaFutureAction<java.util.List<T>> |
collectAsync() |
static java.util.List<T>[] |
collectPartitions(int[] partitionIds) |
<C> JavaPairRDD<K,C> |
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners)
Simplified version of combineByKey that hash-partitions the resulting RDD using the existing
partitioner/parallelism level and using map-side aggregation.
|
<C> JavaPairRDD<K,C> |
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
int numPartitions)
Simplified version of combineByKey that hash-partitions the output RDD and uses map-side
aggregation.
|
<C> JavaPairRDD<K,C> |
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
Partitioner partitioner)
Generic function to combine the elements for each key using a custom set of aggregation
functions.
|
<C> JavaPairRDD<K,C> |
combineByKey(Function<V,C> createCombiner,
Function2<C,V,C> mergeValue,
Function2<C,C,C> mergeCombiners,
Partitioner partitioner,
boolean mapSideCombine,
Serializer serializer)
Generic function to combine the elements for each key using a custom set of aggregation
functions.
|
static SparkContext |
context() |
static long |
count() |
static PartialResult<BoundedDouble> |
countApprox(long timeout) |
static PartialResult<BoundedDouble> |
countApprox(long timeout,
double confidence) |
static long |
countApproxDistinct(double relativeSD) |
JavaPairRDD<K,java.lang.Long> |
countApproxDistinctByKey(double relativeSD)
Return approximate number of distinct values for each key in this RDD.
|
JavaPairRDD<K,java.lang.Long> |
countApproxDistinctByKey(double relativeSD,
int numPartitions)
Return approximate number of distinct values for each key in this RDD.
|
JavaPairRDD<K,java.lang.Long> |
countApproxDistinctByKey(double relativeSD,
Partitioner partitioner)
Return approximate number of distinct values for each key in this RDD.
|
static JavaFutureAction<java.lang.Long> |
countAsync() |
java.util.Map<K,java.lang.Long> |
countByKey()
Count the number of elements for each key, and return the result to the master as a Map.
|
PartialResult<java.util.Map<K,BoundedDouble>> |
countByKeyApprox(long timeout)
Approximate version of countByKey that can return a partial result if it does
not finish within a timeout.
|
PartialResult<java.util.Map<K,BoundedDouble>> |
countByKeyApprox(long timeout,
double confidence)
Approximate version of countByKey that can return a partial result if it does
not finish within a timeout.
|
static java.util.Map<T,java.lang.Long> |
countByValue() |
static PartialResult<java.util.Map<T,BoundedDouble>> |
countByValueApprox(long timeout) |
static PartialResult<java.util.Map<T,BoundedDouble>> |
countByValueApprox(long timeout,
double confidence) |
JavaPairRDD<K,V> |
distinct()
Return a new RDD containing the distinct elements in this RDD.
|
JavaPairRDD<K,V> |
distinct(int numPartitions)
Return a new RDD containing the distinct elements in this RDD.
|
JavaPairRDD<K,V> |
filter(Function<scala.Tuple2<K,V>,java.lang.Boolean> f)
Return a new RDD containing only the elements that satisfy a predicate.
|
scala.Tuple2<K,V> |
first()
Return the first element in this RDD.
|
static <U> JavaRDD<U> |
flatMap(FlatMapFunction<T,U> f) |
static JavaDoubleRDD |
flatMapToDouble(DoubleFlatMapFunction<T> f) |
static <K2,V2> JavaPairRDD<K2,V2> |
flatMapToPair(PairFlatMapFunction<T,K2,V2> f) |
<U> JavaPairRDD<K,U> |
flatMapValues(Function<V,java.lang.Iterable<U>> f)
Pass each value in the key-value pair RDD through a flatMap function without changing the
keys; this also retains the original RDD's partitioning.
|
static T |
fold(T zeroValue,
Function2<T,T,T> f) |
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value"
which may be added to the result an arbitrary number of times, and must not change the result
(e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
|
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
int numPartitions,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value" which
may be added to the result an arbitrary number of times, and must not change the result
(e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
|
JavaPairRDD<K,V> |
foldByKey(V zeroValue,
Partitioner partitioner,
Function2<V,V,V> func)
Merge the values for each key using an associative function and a neutral "zero value" which
may be added to the result an arbitrary number of times, and must not change the result
(e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
|
static void |
foreach(VoidFunction<T> f) |
static JavaFutureAction<java.lang.Void> |
foreachAsync(VoidFunction<T> f) |
static void |
foreachPartition(VoidFunction<java.util.Iterator<T>> f) |
static JavaFutureAction<java.lang.Void> |
foreachPartitionAsync(VoidFunction<java.util.Iterator<T>> f) |
static <K,V> JavaPairRDD<K,V> |
fromJavaRDD(JavaRDD<scala.Tuple2<K,V>> rdd)
Convert a JavaRDD of key-value pairs to JavaPairRDD.
|
static <K,V> JavaPairRDD<K,V> |
fromRDD(RDD<scala.Tuple2<K,V>> rdd,
scala.reflect.ClassTag<K> evidence$5,
scala.reflect.ClassTag<V> evidence$6) |
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> |
fullOuterJoin(JavaPairRDD<K,W> other)
Perform a full outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> |
fullOuterJoin(JavaPairRDD<K,W> other,
int numPartitions)
Perform a full outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> |
fullOuterJoin(JavaPairRDD<K,W> other,
Partitioner partitioner)
Perform a full outer join of
this and other . |
static Optional<java.lang.String> |
getCheckpointFile() |
static int |
getNumPartitions() |
static StorageLevel |
getStorageLevel() |
static JavaRDD<java.util.List<T>> |
glom() |
static <U> JavaPairRDD<U,java.lang.Iterable<T>> |
groupBy(Function<T,U> f) |
static <U> JavaPairRDD<U,java.lang.Iterable<T>> |
groupBy(Function<T,U> f,
int numPartitions) |
JavaPairRDD<K,java.lang.Iterable<V>> |
groupByKey()
Group the values for each key in the RDD into a single sequence.
|
JavaPairRDD<K,java.lang.Iterable<V>> |
groupByKey(int numPartitions)
Group the values for each key in the RDD into a single sequence.
|
JavaPairRDD<K,java.lang.Iterable<V>> |
groupByKey(Partitioner partitioner)
Group the values for each key in the RDD into a single sequence.
|
<W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> |
groupWith(JavaPairRDD<K,W> other)
Alias for cogroup.
|
<W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> |
groupWith(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2)
Alias for cogroup.
|
<W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> |
groupWith(JavaPairRDD<K,W1> other1,
JavaPairRDD<K,W2> other2,
JavaPairRDD<K,W3> other3)
Alias for cogroup.
|
static int |
id() |
JavaPairRDD<K,V> |
intersection(JavaPairRDD<K,V> other)
Return the intersection of this RDD and another one.
|
static boolean |
isCheckpointed() |
static boolean |
isEmpty() |
static java.util.Iterator<T> |
iterator(Partition split,
TaskContext taskContext) |
<W> JavaPairRDD<K,scala.Tuple2<V,W>> |
join(JavaPairRDD<K,W> other)
Return an RDD containing all pairs of elements with matching keys in
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<V,W>> |
join(JavaPairRDD<K,W> other,
int numPartitions)
Return an RDD containing all pairs of elements with matching keys in
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<V,W>> |
join(JavaPairRDD<K,W> other,
Partitioner partitioner)
Return an RDD containing all pairs of elements with matching keys in
this and other . |
scala.reflect.ClassTag<K> |
kClassTag() |
static <U> JavaPairRDD<U,T> |
keyBy(Function<T,U> f) |
JavaRDD<K> |
keys()
Return an RDD with the keys of each tuple.
|
<W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> |
leftOuterJoin(JavaPairRDD<K,W> other)
Perform a left outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> |
leftOuterJoin(JavaPairRDD<K,W> other,
int numPartitions)
Perform a left outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> |
leftOuterJoin(JavaPairRDD<K,W> other,
Partitioner partitioner)
Perform a left outer join of
this and other . |
java.util.List<V> |
lookup(K key)
Return the list of values in the RDD for key
key . |
static <R> JavaRDD<R> |
map(Function<T,R> f) |
static <U> JavaRDD<U> |
mapPartitions(FlatMapFunction<java.util.Iterator<T>,U> f) |
static <U> JavaRDD<U> |
mapPartitions(FlatMapFunction<java.util.Iterator<T>,U> f,
boolean preservesPartitioning) |
static JavaDoubleRDD |
mapPartitionsToDouble(DoubleFlatMapFunction<java.util.Iterator<T>> f) |
static JavaDoubleRDD |
mapPartitionsToDouble(DoubleFlatMapFunction<java.util.Iterator<T>> f,
boolean preservesPartitioning) |
static <K2,V2> JavaPairRDD<K2,V2> |
mapPartitionsToPair(PairFlatMapFunction<java.util.Iterator<T>,K2,V2> f) |
static <K2,V2> JavaPairRDD<K2,V2> |
mapPartitionsToPair(PairFlatMapFunction<java.util.Iterator<T>,K2,V2> f,
boolean preservesPartitioning) |
static <R> JavaRDD<R> |
mapPartitionsWithIndex(Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<R>> f,
boolean preservesPartitioning) |
static <R> boolean |
mapPartitionsWithIndex$default$2() |
static <R> JavaDoubleRDD |
mapToDouble(DoubleFunction<T> f) |
static <K2,V2> JavaPairRDD<K2,V2> |
mapToPair(PairFunction<T,K2,V2> f) |
<U> JavaPairRDD<K,U> |
mapValues(Function<V,U> f)
Pass each value in the key-value pair RDD through a map function without changing the keys;
this also retains the original RDD's partitioning.
|
static T |
max(java.util.Comparator<T> comp) |
static T |
min(java.util.Comparator<T> comp) |
static java.lang.String |
name() |
JavaPairRDD<K,V> |
partitionBy(Partitioner partitioner)
Return a copy of the RDD partitioned using the specified partitioner.
|
static Optional<Partitioner> |
partitioner() |
static java.util.List<Partition> |
partitions() |
JavaPairRDD<K,V> |
persist(StorageLevel newLevel)
Set this RDD's storage level to persist its values across operations after the first time
it is computed.
|
static JavaRDD<java.lang.String> |
pipe(java.util.List<java.lang.String> command) |
static JavaRDD<java.lang.String> |
pipe(java.util.List<java.lang.String> command,
java.util.Map<java.lang.String,java.lang.String> env) |
static JavaRDD<java.lang.String> |
pipe(java.util.List<java.lang.String> command,
java.util.Map<java.lang.String,java.lang.String> env,
boolean separateWorkingDir,
int bufferSize) |
static JavaRDD<java.lang.String> |
pipe(java.lang.String command) |
RDD<scala.Tuple2<K,V>> |
rdd() |
static T |
reduce(Function2<T,T,T> f) |
JavaPairRDD<K,V> |
reduceByKey(Function2<V,V,V> func)
Merge the values for each key using an associative and commutative reduce function.
|
JavaPairRDD<K,V> |
reduceByKey(Function2<V,V,V> func,
int numPartitions)
Merge the values for each key using an associative and commutative reduce function.
|
JavaPairRDD<K,V> |
reduceByKey(Partitioner partitioner,
Function2<V,V,V> func)
Merge the values for each key using an associative and commutative reduce function.
|
java.util.Map<K,V> |
reduceByKeyLocally(Function2<V,V,V> func)
Merge the values for each key using an associative and commutative reduce function, but return
the result immediately to the master as a Map.
|
JavaPairRDD<K,V> |
repartition(int numPartitions)
Return a new RDD that has exactly numPartitions partitions.
|
JavaPairRDD<K,V> |
repartitionAndSortWithinPartitions(Partitioner partitioner)
Repartition the RDD according to the given partitioner and, within each resulting partition,
sort records by their keys.
|
JavaPairRDD<K,V> |
repartitionAndSortWithinPartitions(Partitioner partitioner,
java.util.Comparator<K> comp)
Repartition the RDD according to the given partitioner and, within each resulting partition,
sort records by their keys.
|
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> |
rightOuterJoin(JavaPairRDD<K,W> other)
Perform a right outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> |
rightOuterJoin(JavaPairRDD<K,W> other,
int numPartitions)
Perform a right outer join of
this and other . |
<W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> |
rightOuterJoin(JavaPairRDD<K,W> other,
Partitioner partitioner)
Perform a right outer join of
this and other . |
JavaPairRDD<K,V> |
sample(boolean withReplacement,
double fraction)
Return a sampled subset of this RDD.
|
JavaPairRDD<K,V> |
sample(boolean withReplacement,
double fraction,
long seed)
Return a sampled subset of this RDD.
|
JavaPairRDD<K,V> |
sampleByKey(boolean withReplacement,
java.util.Map<K,java.lang.Double> fractions)
Return a subset of this RDD sampled by key (via stratified sampling).
|
JavaPairRDD<K,V> |
sampleByKey(boolean withReplacement,
java.util.Map<K,java.lang.Double> fractions,
long seed)
Return a subset of this RDD sampled by key (via stratified sampling).
|
JavaPairRDD<K,V> |
sampleByKeyExact(boolean withReplacement,
java.util.Map<K,java.lang.Double> fractions)
Return a subset of this RDD sampled by key (via stratified sampling) containing exactly
math.ceil(numItems * samplingRate) for each stratum (group of pairs with the same key).
|
JavaPairRDD<K,V> |
sampleByKeyExact(boolean withReplacement,
java.util.Map<K,java.lang.Double> fractions,
long seed)
Return a subset of this RDD sampled by key (via stratified sampling) containing exactly
math.ceil(numItems * samplingRate) for each stratum (group of pairs with the same key).
|
void |
saveAsHadoopDataset(org.apache.hadoop.mapred.JobConf conf)
Output the RDD to any Hadoop-supported storage system, using a Hadoop JobConf object for
that storage system.
|
<F extends org.apache.hadoop.mapred.OutputFormat<?,?>> |
saveAsHadoopFile(java.lang.String path,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass,
java.lang.Class<F> outputFormatClass)
Output the RDD to any Hadoop-supported file system.
|
<F extends org.apache.hadoop.mapred.OutputFormat<?,?>> |
saveAsHadoopFile(java.lang.String path,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass,
java.lang.Class<F> outputFormatClass,
java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)
Output the RDD to any Hadoop-supported file system, compressing with the supplied codec.
|
<F extends org.apache.hadoop.mapred.OutputFormat<?,?>> |
saveAsHadoopFile(java.lang.String path,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass,
java.lang.Class<F> outputFormatClass,
org.apache.hadoop.mapred.JobConf conf)
Output the RDD to any Hadoop-supported file system.
|
void |
saveAsNewAPIHadoopDataset(org.apache.hadoop.conf.Configuration conf)
Output the RDD to any Hadoop-supported storage system, using
a Configuration object for that storage system.
|
<F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> |
saveAsNewAPIHadoopFile(java.lang.String path,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass,
java.lang.Class<F> outputFormatClass)
Output the RDD to any Hadoop-supported file system.
|
<F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> |
saveAsNewAPIHadoopFile(java.lang.String path,
java.lang.Class<?> keyClass,
java.lang.Class<?> valueClass,
java.lang.Class<F> outputFormatClass,
org.apache.hadoop.conf.Configuration conf)
Output the RDD to any Hadoop-supported file system.
|
static void |
saveAsObjectFile(java.lang.String path) |
static void |
saveAsTextFile(java.lang.String path) |
static void |
saveAsTextFile(java.lang.String path,
java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec) |
JavaPairRDD<K,V> |
setName(java.lang.String name)
Assign a name to this RDD
|
JavaPairRDD<K,V> |
sortByKey()
Sort the RDD by key, so that each partition contains a sorted range of the elements in
ascending order.
|
JavaPairRDD<K,V> |
sortByKey(boolean ascending)
Sort the RDD by key, so that each partition contains a sorted range of the elements.
|
JavaPairRDD<K,V> |
sortByKey(boolean ascending,
int numPartitions)
Sort the RDD by key, so that each partition contains a sorted range of the elements.
|
JavaPairRDD<K,V> |
sortByKey(java.util.Comparator<K> comp)
Sort the RDD by key, so that each partition contains a sorted range of the elements.
|
JavaPairRDD<K,V> |
sortByKey(java.util.Comparator<K> comp,
boolean ascending)
Sort the RDD by key, so that each partition contains a sorted range of the elements.
|
JavaPairRDD<K,V> |
sortByKey(java.util.Comparator<K> comp,
boolean ascending,
int numPartitions)
Sort the RDD by key, so that each partition contains a sorted range of the elements.
|
JavaPairRDD<K,V> |
subtract(JavaPairRDD<K,V> other)
Return an RDD with the elements from
this that are not in other . |
JavaPairRDD<K,V> |
subtract(JavaPairRDD<K,V> other,
int numPartitions)
Return an RDD with the elements from
this that are not in other . |
JavaPairRDD<K,V> |
subtract(JavaPairRDD<K,V> other,
Partitioner p)
Return an RDD with the elements from
this that are not in other . |
<W> JavaPairRDD<K,V> |
subtractByKey(JavaPairRDD<K,W> other)
Return an RDD with the pairs from
this whose keys are not in other . |
<W> JavaPairRDD<K,V> |
subtractByKey(JavaPairRDD<K,W> other,
int numPartitions)
Return an RDD with the pairs from `this` whose keys are not in `other`.
|
<W> JavaPairRDD<K,V> |
subtractByKey(JavaPairRDD<K,W> other,
Partitioner p)
Return an RDD with the pairs from `this` whose keys are not in `other`.
|
static java.util.List<T> |
take(int num) |
static JavaFutureAction<java.util.List<T>> |
takeAsync(int num) |
static java.util.List<T> |
takeOrdered(int num) |
static java.util.List<T> |
takeOrdered(int num,
java.util.Comparator<T> comp) |
static java.util.List<T> |
takeSample(boolean withReplacement,
int num) |
static java.util.List<T> |
takeSample(boolean withReplacement,
int num,
long seed) |
static java.lang.String |
toDebugString() |
static java.util.Iterator<T> |
toLocalIterator() |
static java.util.List<T> |
top(int num) |
static java.util.List<T> |
top(int num,
java.util.Comparator<T> comp) |
static <K,V> RDD<scala.Tuple2<K,V>> |
toRDD(JavaPairRDD<K,V> rdd) |
static <U> U |
treeAggregate(U zeroValue,
Function2<U,T,U> seqOp,
Function2<U,U,U> combOp) |
static <U> U |
treeAggregate(U zeroValue,
Function2<U,T,U> seqOp,
Function2<U,U,U> combOp,
int depth) |
static T |
treeReduce(Function2<T,T,T> f) |
static T |
treeReduce(Function2<T,T,T> f,
int depth) |
JavaPairRDD<K,V> |
union(JavaPairRDD<K,V> other)
Return the union of this RDD and another one.
|
JavaPairRDD<K,V> |
unpersist()
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
|
JavaPairRDD<K,V> |
unpersist(boolean blocking)
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
|
JavaRDD<V> |
values()
Return an RDD with the values of each tuple.
|
scala.reflect.ClassTag<V> |
vClassTag() |
JavaPairRDD<K,V> |
wrapRDD(RDD<scala.Tuple2<K,V>> rdd) |
static <U> JavaPairRDD<T,U> |
zip(JavaRDDLike<U,?> other) |
static <U,V> JavaRDD<V> |
zipPartitions(JavaRDDLike<U,?> other,
FlatMapFunction2<java.util.Iterator<T>,java.util.Iterator<U>,V> f) |
static JavaPairRDD<T,java.lang.Long> |
zipWithIndex() |
static JavaPairRDD<T,java.lang.Long> |
zipWithUniqueId() |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
aggregate, cartesian, checkpoint, collect, collectAsync, collectPartitions, context, count, countApprox, countApprox, countApproxDistinct, countAsync, countByValue, countByValueApprox, countByValueApprox, flatMap, flatMapToDouble, flatMapToPair, fold, foreach, foreachAsync, foreachPartition, foreachPartitionAsync, getCheckpointFile, getNumPartitions, getStorageLevel, glom, groupBy, groupBy, id, isCheckpointed, isEmpty, iterator, keyBy, map, mapPartitions, mapPartitions, mapPartitionsToDouble, mapPartitionsToDouble, mapPartitionsToPair, mapPartitionsToPair, mapPartitionsWithIndex, mapToDouble, mapToPair, max, min, name, partitioner, partitions, pipe, pipe, pipe, pipe, reduce, saveAsObjectFile, saveAsTextFile, saveAsTextFile, take, takeAsync, takeOrdered, takeOrdered, takeSample, takeSample, toDebugString, toLocalIterator, top, top, treeAggregate, treeAggregate, treeReduce, treeReduce, zip, zipPartitions, zipWithIndex, zipWithUniqueId
public static <K,V> JavaPairRDD<K,V> fromRDD(RDD<scala.Tuple2<K,V>> rdd, scala.reflect.ClassTag<K> evidence$5, scala.reflect.ClassTag<V> evidence$6)
public static <K,V> RDD<scala.Tuple2<K,V>> toRDD(JavaPairRDD<K,V> rdd)
public static <K,V> JavaPairRDD<K,V> fromJavaRDD(JavaRDD<scala.Tuple2<K,V>> rdd)
public static java.util.List<Partition> partitions()
public static int getNumPartitions()
public static Optional<Partitioner> partitioner()
public static SparkContext context()
public static int id()
public static StorageLevel getStorageLevel()
public static java.util.Iterator<T> iterator(Partition split, TaskContext taskContext)
public static <R> JavaRDD<R> mapPartitionsWithIndex(Function2<java.lang.Integer,java.util.Iterator<T>,java.util.Iterator<R>> f, boolean preservesPartitioning)
public static <R> JavaDoubleRDD mapToDouble(DoubleFunction<T> f)
public static <K2,V2> JavaPairRDD<K2,V2> mapToPair(PairFunction<T,K2,V2> f)
public static <U> JavaRDD<U> flatMap(FlatMapFunction<T,U> f)
public static JavaDoubleRDD flatMapToDouble(DoubleFlatMapFunction<T> f)
public static <K2,V2> JavaPairRDD<K2,V2> flatMapToPair(PairFlatMapFunction<T,K2,V2> f)
public static <U> JavaRDD<U> mapPartitions(FlatMapFunction<java.util.Iterator<T>,U> f)
public static <U> JavaRDD<U> mapPartitions(FlatMapFunction<java.util.Iterator<T>,U> f, boolean preservesPartitioning)
public static JavaDoubleRDD mapPartitionsToDouble(DoubleFlatMapFunction<java.util.Iterator<T>> f)
public static <K2,V2> JavaPairRDD<K2,V2> mapPartitionsToPair(PairFlatMapFunction<java.util.Iterator<T>,K2,V2> f)
public static JavaDoubleRDD mapPartitionsToDouble(DoubleFlatMapFunction<java.util.Iterator<T>> f, boolean preservesPartitioning)
public static <K2,V2> JavaPairRDD<K2,V2> mapPartitionsToPair(PairFlatMapFunction<java.util.Iterator<T>,K2,V2> f, boolean preservesPartitioning)
public static void foreachPartition(VoidFunction<java.util.Iterator<T>> f)
public static JavaRDD<java.util.List<T>> glom()
public static <U> JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)
public static <U> JavaPairRDD<U,java.lang.Iterable<T>> groupBy(Function<T,U> f)
public static <U> JavaPairRDD<U,java.lang.Iterable<T>> groupBy(Function<T,U> f, int numPartitions)
public static JavaRDD<java.lang.String> pipe(java.lang.String command)
public static JavaRDD<java.lang.String> pipe(java.util.List<java.lang.String> command)
public static JavaRDD<java.lang.String> pipe(java.util.List<java.lang.String> command, java.util.Map<java.lang.String,java.lang.String> env)
public static JavaRDD<java.lang.String> pipe(java.util.List<java.lang.String> command, java.util.Map<java.lang.String,java.lang.String> env, boolean separateWorkingDir, int bufferSize)
public static <U> JavaPairRDD<T,U> zip(JavaRDDLike<U,?> other)
public static <U,V> JavaRDD<V> zipPartitions(JavaRDDLike<U,?> other, FlatMapFunction2<java.util.Iterator<T>,java.util.Iterator<U>,V> f)
public static JavaPairRDD<T,java.lang.Long> zipWithUniqueId()
public static JavaPairRDD<T,java.lang.Long> zipWithIndex()
public static void foreach(VoidFunction<T> f)
public static java.util.List<T> collect()
public static java.util.Iterator<T> toLocalIterator()
public static java.util.List<T>[] collectPartitions(int[] partitionIds)
public static T reduce(Function2<T,T,T> f)
public static T treeReduce(Function2<T,T,T> f, int depth)
public static T treeReduce(Function2<T,T,T> f)
public static T fold(T zeroValue, Function2<T,T,T> f)
public static <U> U aggregate(U zeroValue, Function2<U,T,U> seqOp, Function2<U,U,U> combOp)
public static <U> U treeAggregate(U zeroValue, Function2<U,T,U> seqOp, Function2<U,U,U> combOp, int depth)
public static <U> U treeAggregate(U zeroValue, Function2<U,T,U> seqOp, Function2<U,U,U> combOp)
public static long count()
public static PartialResult<BoundedDouble> countApprox(long timeout, double confidence)
public static PartialResult<BoundedDouble> countApprox(long timeout)
public static java.util.Map<T,java.lang.Long> countByValue()
public static PartialResult<java.util.Map<T,BoundedDouble>> countByValueApprox(long timeout, double confidence)
public static PartialResult<java.util.Map<T,BoundedDouble>> countByValueApprox(long timeout)
public static java.util.List<T> take(int num)
public static java.util.List<T> takeSample(boolean withReplacement, int num)
public static java.util.List<T> takeSample(boolean withReplacement, int num, long seed)
public static boolean isEmpty()
public static void saveAsTextFile(java.lang.String path)
public static void saveAsTextFile(java.lang.String path, java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)
public static void saveAsObjectFile(java.lang.String path)
public static <U> JavaPairRDD<U,T> keyBy(Function<T,U> f)
public static void checkpoint()
public static boolean isCheckpointed()
public static Optional<java.lang.String> getCheckpointFile()
public static java.lang.String toDebugString()
public static java.util.List<T> top(int num, java.util.Comparator<T> comp)
public static java.util.List<T> top(int num)
public static java.util.List<T> takeOrdered(int num, java.util.Comparator<T> comp)
public static T max(java.util.Comparator<T> comp)
public static T min(java.util.Comparator<T> comp)
public static java.util.List<T> takeOrdered(int num)
public static long countApproxDistinct(double relativeSD)
public static java.lang.String name()
public static JavaFutureAction<java.lang.Long> countAsync()
public static JavaFutureAction<java.util.List<T>> collectAsync()
public static JavaFutureAction<java.util.List<T>> takeAsync(int num)
public static JavaFutureAction<java.lang.Void> foreachAsync(VoidFunction<T> f)
public static JavaFutureAction<java.lang.Void> foreachPartitionAsync(VoidFunction<java.util.Iterator<T>> f)
public static <R> boolean mapPartitionsWithIndex$default$2()
public scala.reflect.ClassTag<K> kClassTag()
public scala.reflect.ClassTag<V> vClassTag()
public JavaPairRDD<K,V> cache()
public JavaPairRDD<K,V> persist(StorageLevel newLevel)
newLevel
- (undocumented)public JavaPairRDD<K,V> unpersist()
public JavaPairRDD<K,V> unpersist(boolean blocking)
blocking
- Whether to block until all blocks are deleted.public JavaPairRDD<K,V> distinct()
public JavaPairRDD<K,V> distinct(int numPartitions)
numPartitions
- (undocumented)public JavaPairRDD<K,V> filter(Function<scala.Tuple2<K,V>,java.lang.Boolean> f)
f
- (undocumented)public JavaPairRDD<K,V> coalesce(int numPartitions)
numPartitions
partitions.numPartitions
- (undocumented)public JavaPairRDD<K,V> coalesce(int numPartitions, boolean shuffle)
numPartitions
partitions.numPartitions
- (undocumented)shuffle
- (undocumented)public JavaPairRDD<K,V> repartition(int numPartitions)
Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.
If you are decreasing the number of partitions in this RDD, consider using coalesce
,
which can avoid performing a shuffle.
numPartitions
- (undocumented)public JavaPairRDD<K,V> sample(boolean withReplacement, double fraction)
withReplacement
- (undocumented)fraction
- (undocumented)public JavaPairRDD<K,V> sample(boolean withReplacement, double fraction, long seed)
withReplacement
- (undocumented)fraction
- (undocumented)seed
- (undocumented)public JavaPairRDD<K,V> sampleByKey(boolean withReplacement, java.util.Map<K,java.lang.Double> fractions, long seed)
Create a sample of this RDD using variable sampling rates for different keys as specified by
fractions
, a key to sampling rate map, via simple random sampling with one pass over the
RDD, to produce a sample of size that's approximately equal to the sum of
math.ceil(numItems * samplingRate) over all key values.
withReplacement
- (undocumented)fractions
- (undocumented)seed
- (undocumented)public JavaPairRDD<K,V> sampleByKey(boolean withReplacement, java.util.Map<K,java.lang.Double> fractions)
Create a sample of this RDD using variable sampling rates for different keys as specified by
fractions
, a key to sampling rate map, via simple random sampling with one pass over the
RDD, to produce a sample of size that's approximately equal to the sum of
math.ceil(numItems * samplingRate) over all key values.
Use Utils.random.nextLong as the default seed for the random number generator.
withReplacement
- (undocumented)fractions
- (undocumented)public JavaPairRDD<K,V> sampleByKeyExact(boolean withReplacement, java.util.Map<K,java.lang.Double> fractions, long seed)
This method differs from sampleByKey
in that we make additional passes over the RDD to
create a sample size that's exactly equal to the sum of math.ceil(numItems * samplingRate)
over all key values with a 99.99% confidence. When sampling without replacement, we need one
additional pass over the RDD to guarantee sample size; when sampling with replacement, we need
two additional passes.
withReplacement
- (undocumented)fractions
- (undocumented)seed
- (undocumented)public JavaPairRDD<K,V> sampleByKeyExact(boolean withReplacement, java.util.Map<K,java.lang.Double> fractions)
This method differs from sampleByKey
in that we make additional passes over the RDD to
create a sample size that's exactly equal to the sum of math.ceil(numItems * samplingRate)
over all key values with a 99.99% confidence. When sampling without replacement, we need one
additional pass over the RDD to guarantee sample size; when sampling with replacement, we need
two additional passes.
Use Utils.random.nextLong as the default seed for the random number generator.
withReplacement
- (undocumented)fractions
- (undocumented)public JavaPairRDD<K,V> union(JavaPairRDD<K,V> other)
.distinct()
to eliminate them).other
- (undocumented)public JavaPairRDD<K,V> intersection(JavaPairRDD<K,V> other)
Note that this method performs a shuffle internally.
other
- (undocumented)public scala.Tuple2<K,V> first()
JavaRDDLike
public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, Partitioner partitioner, boolean mapSideCombine, Serializer serializer)
- createCombiner
, which turns a V into a C (e.g., creates a one-element list)
- mergeValue
, to merge a V into a C (e.g., adds it to the end of a list)
- mergeCombiners
, to combine two C's into a single one.
In addition, users can control the partitioning of the output RDD, the serializer that is use for the shuffle, and whether to perform map-side aggregation (if a mapper can produce multiple items with the same key).
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)partitioner
- (undocumented)mapSideCombine
- (undocumented)serializer
- (undocumented)public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, Partitioner partitioner)
- createCombiner
, which turns a V into a C (e.g., creates a one-element list)
- mergeValue
, to merge a V into a C (e.g., adds it to the end of a list)
- mergeCombiners
, to combine two C's into a single one.
In addition, users can control the partitioning of the output RDD. This method automatically uses map-side aggregation in shuffling the RDD.
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)partitioner
- (undocumented)public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners, int numPartitions)
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)numPartitions
- (undocumented)public JavaPairRDD<K,V> reduceByKey(Partitioner partitioner, Function2<V,V,V> func)
partitioner
- (undocumented)func
- (undocumented)public java.util.Map<K,V> reduceByKeyLocally(Function2<V,V,V> func)
func
- (undocumented)public java.util.Map<K,java.lang.Long> countByKey()
public PartialResult<java.util.Map<K,BoundedDouble>> countByKeyApprox(long timeout)
timeout
- (undocumented)public PartialResult<java.util.Map<K,BoundedDouble>> countByKeyApprox(long timeout, double confidence)
timeout
- (undocumented)confidence
- (undocumented)public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, Partitioner partitioner, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)partitioner
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, int numPartitions, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)numPartitions
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)public <U> JavaPairRDD<K,U> aggregateByKey(U zeroValue, Function2<U,V,U> seqFunc, Function2<U,U,U> combFunc)
zeroValue
- (undocumented)seqFunc
- (undocumented)combFunc
- (undocumented)public JavaPairRDD<K,V> foldByKey(V zeroValue, Partitioner partitioner, Function2<V,V,V> func)
zeroValue
- (undocumented)partitioner
- (undocumented)func
- (undocumented)public JavaPairRDD<K,V> foldByKey(V zeroValue, int numPartitions, Function2<V,V,V> func)
zeroValue
- (undocumented)numPartitions
- (undocumented)func
- (undocumented)public JavaPairRDD<K,V> foldByKey(V zeroValue, Function2<V,V,V> func)
zeroValue
- (undocumented)func
- (undocumented)public JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func, int numPartitions)
func
- (undocumented)numPartitions
- (undocumented)public JavaPairRDD<K,java.lang.Iterable<V>> groupByKey(Partitioner partitioner)
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
partitioner
- (undocumented)public JavaPairRDD<K,java.lang.Iterable<V>> groupByKey(int numPartitions)
numPartitions
partitions.
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
numPartitions
- (undocumented)public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other)
this
that are not in other
.
Uses this
partitioner/partition size, because even if other
is huge, the resulting
RDD will be <= us.
other
- (undocumented)public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other, int numPartitions)
this
that are not in other
.other
- (undocumented)numPartitions
- (undocumented)public JavaPairRDD<K,V> subtract(JavaPairRDD<K,V> other, Partitioner p)
this
that are not in other
.other
- (undocumented)p
- (undocumented)public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other)
this
whose keys are not in other
.
Uses this
partitioner/partition size, because even if other
is huge, the resulting
RDD will be <= us.
other
- (undocumented)public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other, int numPartitions)
public <W> JavaPairRDD<K,V> subtractByKey(JavaPairRDD<K,W> other, Partitioner p)
public JavaPairRDD<K,V> partitionBy(Partitioner partitioner)
partitioner
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. Each
pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this
and
(k, v2) is in other
. Uses the given Partitioner to partition the output RDD.other
- (undocumented)partitioner
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Uses the given Partitioner to
partition the output RDD.other
- (undocumented)partitioner
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Uses the given Partitioner to
partition the output RDD.other
- (undocumented)partitioner
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other, Partitioner partitioner)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Uses the given Partitioner to partition the output RDD.other
- (undocumented)partitioner
- (undocumented)public <C> JavaPairRDD<K,C> combineByKey(Function<V,C> createCombiner, Function2<C,V,C> mergeValue, Function2<C,C,C> mergeCombiners)
createCombiner
- (undocumented)mergeValue
- (undocumented)mergeCombiners
- (undocumented)public JavaPairRDD<K,V> reduceByKey(Function2<V,V,V> func)
func
- (undocumented)public JavaPairRDD<K,java.lang.Iterable<V>> groupByKey()
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over
each key, using JavaPairRDD.reduceByKey
or JavaPairRDD.combineByKey
will provide much better performance.
public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other)
this
and other
. Each
pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this
and
(k, v2) is in other
. Performs a hash join across the cluster.other
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<V,W>> join(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. Each
pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this
and
(k, v2) is in other
. Performs a hash join across the cluster.other
- (undocumented)numPartitions
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Hash-partitions the output
using the existing partitioner/parallelism level.other
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<V,Optional<W>>> leftOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (v, Some(w))) for w in other
, or the
pair (k, (v, None)) if no elements in other
have key k. Hash-partitions the output
into numPartitions
partitions.other
- (undocumented)numPartitions
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Hash-partitions the resulting
RDD using the existing partitioner/parallelism level.other
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,W>> rightOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, w) in other
, the
resulting RDD will either contain all pairs (k, (Some(v), w)) for v in this
, or the
pair (k, (None, w)) if no elements in this
have key k. Hash-partitions the resulting
RDD into the given number of partitions.other
- (undocumented)numPartitions
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Hash-partitions the resulting RDD using the existing partitioner/
parallelism level.other
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<Optional<V>,Optional<W>>> fullOuterJoin(JavaPairRDD<K,W> other, int numPartitions)
this
and other
. For each element (k, v) in this
, the
resulting RDD will either contain all pairs (k, (Some(v), Some(w))) for w in other
, or
the pair (k, (Some(v), None)) if no elements in other
have key k. Similarly, for each
element (k, w) in other
, the resulting RDD will either contain all pairs
(k, (Some(v), Some(w))) for v in this
, or the pair (k, (None, Some(w))) if no elements
in this
have key k. Hash-partitions the resulting RDD into the given number of partitions.other
- (undocumented)numPartitions
- (undocumented)public java.util.Map<K,V> collectAsMap()
public <U> JavaPairRDD<K,U> mapValues(Function<V,U> f)
f
- (undocumented)public <U> JavaPairRDD<K,U> flatMapValues(Function<V,java.lang.Iterable<U>> f)
f
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> cogroup(JavaPairRDD<K,W> other, Partitioner partitioner)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.other
- (undocumented)partitioner
- (undocumented)public <W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, Partitioner partitioner)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.other1
- (undocumented)other2
- (undocumented)partitioner
- (undocumented)public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3, Partitioner partitioner)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)partitioner
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> cogroup(JavaPairRDD<K,W> other)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.other
- (undocumented)public <W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.other1
- (undocumented)other2
- (undocumented)public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> cogroup(JavaPairRDD<K,W> other, int numPartitions)
this
or other
, return a resulting RDD that contains a tuple with the
list of values for that key in this
as well as other
.other
- (undocumented)numPartitions
- (undocumented)public <W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, int numPartitions)
this
or other1
or other2
, return a resulting RDD that contains a
tuple with the list of values for that key in this
, other1
and other2
.other1
- (undocumented)other2
- (undocumented)numPartitions
- (undocumented)public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> cogroup(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3, int numPartitions)
this
or other1
or other2
or other3
,
return a resulting RDD that contains a tuple with the list of values
for that key in this
, other1
, other2
and other3
.other1
- (undocumented)other2
- (undocumented)other3
- (undocumented)numPartitions
- (undocumented)public <W> JavaPairRDD<K,scala.Tuple2<java.lang.Iterable<V>,java.lang.Iterable<W>>> groupWith(JavaPairRDD<K,W> other)
public <W1,W2> JavaPairRDD<K,scala.Tuple3<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>>> groupWith(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2)
public <W1,W2,W3> JavaPairRDD<K,scala.Tuple4<java.lang.Iterable<V>,java.lang.Iterable<W1>,java.lang.Iterable<W2>,java.lang.Iterable<W3>>> groupWith(JavaPairRDD<K,W1> other1, JavaPairRDD<K,W2> other2, JavaPairRDD<K,W3> other3)
public java.util.List<V> lookup(K key)
key
. This operation is done efficiently if the
RDD has a known partitioner by only searching the partition that the key maps to.key
- (undocumented)public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(java.lang.String path, java.lang.Class<?> keyClass, java.lang.Class<?> valueClass, java.lang.Class<F> outputFormatClass, org.apache.hadoop.mapred.JobConf conf)
public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(java.lang.String path, java.lang.Class<?> keyClass, java.lang.Class<?> valueClass, java.lang.Class<F> outputFormatClass)
public <F extends org.apache.hadoop.mapred.OutputFormat<?,?>> void saveAsHadoopFile(java.lang.String path, java.lang.Class<?> keyClass, java.lang.Class<?> valueClass, java.lang.Class<F> outputFormatClass, java.lang.Class<? extends org.apache.hadoop.io.compress.CompressionCodec> codec)
public <F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> void saveAsNewAPIHadoopFile(java.lang.String path, java.lang.Class<?> keyClass, java.lang.Class<?> valueClass, java.lang.Class<F> outputFormatClass, org.apache.hadoop.conf.Configuration conf)
public void saveAsNewAPIHadoopDataset(org.apache.hadoop.conf.Configuration conf)
conf
- (undocumented)public <F extends org.apache.hadoop.mapreduce.OutputFormat<?,?>> void saveAsNewAPIHadoopFile(java.lang.String path, java.lang.Class<?> keyClass, java.lang.Class<?> valueClass, java.lang.Class<F> outputFormatClass)
public void saveAsHadoopDataset(org.apache.hadoop.mapred.JobConf conf)
conf
- (undocumented)public JavaPairRDD<K,V> repartitionAndSortWithinPartitions(Partitioner partitioner)
This is more efficient than calling repartition
and then sorting within each partition
because it can push the sorting down into the shuffle machinery.
partitioner
- (undocumented)public JavaPairRDD<K,V> repartitionAndSortWithinPartitions(Partitioner partitioner, java.util.Comparator<K> comp)
This is more efficient than calling repartition
and then sorting within each partition
because it can push the sorting down into the shuffle machinery.
partitioner
- (undocumented)comp
- (undocumented)public JavaPairRDD<K,V> sortByKey()
collect
or save
on the resulting RDD will return or output an
ordered list of records (in the save
case, they will be written to multiple part-X
files
in the filesystem, in order of the keys).public JavaPairRDD<K,V> sortByKey(boolean ascending)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).ascending
- (undocumented)public JavaPairRDD<K,V> sortByKey(boolean ascending, int numPartitions)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).ascending
- (undocumented)numPartitions
- (undocumented)public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).comp
- (undocumented)public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp, boolean ascending)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).comp
- (undocumented)ascending
- (undocumented)public JavaPairRDD<K,V> sortByKey(java.util.Comparator<K> comp, boolean ascending, int numPartitions)
collect
or save
on the resulting RDD will return or output an ordered list of records
(in the save
case, they will be written to multiple part-X
files in the filesystem, in
order of the keys).comp
- (undocumented)ascending
- (undocumented)numPartitions
- (undocumented)public JavaRDD<V> values()
public JavaPairRDD<K,java.lang.Long> countApproxDistinctByKey(double relativeSD, Partitioner partitioner)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.partitioner
- partitioner of the resulting RDD.public JavaPairRDD<K,java.lang.Long> countApproxDistinctByKey(double relativeSD, int numPartitions)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.numPartitions
- number of partitions of the resulting RDD.public JavaPairRDD<K,java.lang.Long> countApproxDistinctByKey(double relativeSD)
The algorithm used is based on streamlib's implementation of "HyperLogLog in Practice: Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm", available here.
relativeSD
- Relative accuracy. Smaller values create counters that require more space.
It must be greater than 0.000017.public JavaPairRDD<K,V> setName(java.lang.String name)