Package org.apache.spark.api.java
Class JavaDoubleRDD
Object
org.apache.spark.api.java.JavaDoubleRDD
- All Implemented Interfaces:
Serializable
,JavaRDDLike<Double,
,JavaDoubleRDD> scala.Serializable
- See Also:
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptioncache()
Persist this RDD with the default storage level (MEMORY_ONLY
).scala.reflect.ClassTag<Double>
classTag()
coalesce
(int numPartitions) Return a new RDD that is reduced intonumPartitions
partitions.coalesce
(int numPartitions, boolean shuffle) Return a new RDD that is reduced intonumPartitions
partitions.distinct()
Return a new RDD containing the distinct elements in this RDD.distinct
(int numPartitions) Return a new RDD containing the distinct elements in this RDD.Return a new RDD containing only the elements that satisfy a predicate.first()
Return the first element in this RDD.static JavaDoubleRDD
long[]
histogram
(double[] buckets) Compute a histogram using the provided buckets.scala.Tuple2<double[],
long[]> histogram
(int bucketCount) Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD.long[]
intersection
(JavaDoubleRDD other) Return the intersection of this RDD and another one.max()
Returns the maximum element from this RDD as defined by the default comparator natural order.mean()
Compute the mean of this RDD's elements.meanApprox
(long timeout) Approximate operation to return the mean within a timeout.meanApprox
(long timeout, Double confidence) Return the approximate mean of the elements in this RDD.min()
Returns the minimum element from this RDD as defined by the default comparator natural order.persist
(StorageLevel newLevel) Set this RDD's storage level to persist its values across operations after the first time it is computed.popStdev()
Compute the population standard deviation of this RDD's elements.Compute the population variance of this RDD's elements.rdd()
repartition
(int numPartitions) Return a new RDD that has exactly numPartitions partitions.Return a sampled subset of this RDD.Return a sampled subset of this RDD.Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).Compute the sample variance of this RDD's elements (which corrects for bias in estimating the standard variance by dividing by N-1 instead of N).Assign a name to this RDDsrdd()
stats()
Return aStatCounter
object that captures the mean, variance and count of the RDD's elements in one operation.stdev()
Compute the population standard deviation of this RDD's elements.subtract
(JavaDoubleRDD other) Return an RDD with the elements fromthis
that are not inother
.subtract
(JavaDoubleRDD other, int numPartitions) Return an RDD with the elements fromthis
that are not inother
.subtract
(JavaDoubleRDD other, Partitioner p) Return an RDD with the elements fromthis
that are not inother
.sum()
Add up the elements in this RDD.sumApprox
(long timeout) Approximate operation to return the sum within a timeout.Approximate operation to return the sum within a timeout.toRDD
(JavaDoubleRDD rdd) union
(JavaDoubleRDD other) Return the union of this RDD and another one.Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.unpersist
(boolean blocking) Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.variance()
Compute the population variance of this RDD's elements.Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.spark.api.java.JavaRDDLike
aggregate, cartesian, checkpoint, collect, collectAsync, collectPartitions, context, count, countApprox, countApprox, countApproxDistinct, countAsync, countByValue, countByValueApprox, countByValueApprox, flatMap, flatMapToDouble, flatMapToPair, fold, foreach, foreachAsync, foreachPartition, foreachPartitionAsync, getCheckpointFile, getNumPartitions, getStorageLevel, glom, groupBy, groupBy, id, isCheckpointed, isEmpty, iterator, keyBy, map, mapPartitions, mapPartitions, mapPartitionsToDouble, mapPartitionsToDouble, mapPartitionsToPair, mapPartitionsToPair, mapPartitionsWithIndex, mapToDouble, mapToPair, max, min, name, partitioner, partitions, pipe, pipe, pipe, pipe, pipe, reduce, saveAsObjectFile, saveAsTextFile, saveAsTextFile, take, takeAsync, takeOrdered, takeOrdered, takeSample, takeSample, toDebugString, toLocalIterator, top, top, treeAggregate, treeAggregate, treeAggregate, treeReduce, treeReduce, zip, zipPartitions, zipWithIndex, zipWithUniqueId
-
Constructor Details
-
JavaDoubleRDD
-
-
Method Details
-
fromRDD
-
toRDD
-
srdd
-
classTag
-
rdd
-
wrapRDD
-
cache
Persist this RDD with the default storage level (MEMORY_ONLY
).- Returns:
- (undocumented)
-
persist
Set this RDD's storage level to persist its values across operations after the first time it is computed. Can only be called once on each RDD.- Parameters:
newLevel
- (undocumented)- Returns:
- (undocumented)
-
unpersist
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk. This method blocks until all blocks are deleted.- Returns:
- (undocumented)
-
unpersist
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.- Parameters:
blocking
- Whether to block until all blocks are deleted.- Returns:
- (undocumented)
-
first
Description copied from interface:JavaRDDLike
Return the first element in this RDD.- Returns:
- (undocumented)
-
distinct
Return a new RDD containing the distinct elements in this RDD.- Returns:
- (undocumented)
-
distinct
Return a new RDD containing the distinct elements in this RDD.- Parameters:
numPartitions
- (undocumented)- Returns:
- (undocumented)
-
filter
Return a new RDD containing only the elements that satisfy a predicate.- Parameters:
f
- (undocumented)- Returns:
- (undocumented)
-
coalesce
Return a new RDD that is reduced intonumPartitions
partitions.- Parameters:
numPartitions
- (undocumented)- Returns:
- (undocumented)
-
coalesce
Return a new RDD that is reduced intonumPartitions
partitions.- Parameters:
numPartitions
- (undocumented)shuffle
- (undocumented)- Returns:
- (undocumented)
-
repartition
Return a new RDD that has exactly numPartitions partitions.Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to redistribute data.
If you are decreasing the number of partitions in this RDD, consider using
coalesce
, which can avoid performing a shuffle.- Parameters:
numPartitions
- (undocumented)- Returns:
- (undocumented)
-
subtract
Return an RDD with the elements fromthis
that are not inother
.Uses
this
partitioner/partition size, because even ifother
is huge, the resulting RDD will be <= us.- Parameters:
other
- (undocumented)- Returns:
- (undocumented)
-
subtract
Return an RDD with the elements fromthis
that are not inother
.- Parameters:
other
- (undocumented)numPartitions
- (undocumented)- Returns:
- (undocumented)
-
subtract
Return an RDD with the elements fromthis
that are not inother
.- Parameters:
other
- (undocumented)p
- (undocumented)- Returns:
- (undocumented)
-
sample
Return a sampled subset of this RDD.- Parameters:
withReplacement
- (undocumented)fraction
- (undocumented)- Returns:
- (undocumented)
-
sample
Return a sampled subset of this RDD.- Parameters:
withReplacement
- (undocumented)fraction
- (undocumented)seed
- (undocumented)- Returns:
- (undocumented)
-
union
Return the union of this RDD and another one. Any identical elements will appear multiple times (use.distinct()
to eliminate them).- Parameters:
other
- (undocumented)- Returns:
- (undocumented)
-
intersection
Return the intersection of this RDD and another one. The output will not contain any duplicate elements, even if the input RDDs did.- Parameters:
other
- (undocumented)- Returns:
- (undocumented)
- Note:
- This method performs a shuffle internally.
-
sum
Add up the elements in this RDD. -
min
Returns the minimum element from this RDD as defined by the default comparator natural order.- Returns:
- the minimum of the RDD
-
max
Returns the maximum element from this RDD as defined by the default comparator natural order.- Returns:
- the maximum of the RDD
-
stats
Return aStatCounter
object that captures the mean, variance and count of the RDD's elements in one operation.- Returns:
- (undocumented)
-
mean
Compute the mean of this RDD's elements. -
variance
Compute the population variance of this RDD's elements. -
stdev
Compute the population standard deviation of this RDD's elements. -
sampleStdev
Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).- Returns:
- (undocumented)
-
sampleVariance
Compute the sample variance of this RDD's elements (which corrects for bias in estimating the standard variance by dividing by N-1 instead of N).- Returns:
- (undocumented)
-
popStdev
Compute the population standard deviation of this RDD's elements.- Returns:
- (undocumented)
-
popVariance
Compute the population variance of this RDD's elements.- Returns:
- (undocumented)
-
meanApprox
Return the approximate mean of the elements in this RDD. -
meanApprox
Approximate operation to return the mean within a timeout.- Parameters:
timeout
- (undocumented)- Returns:
- (undocumented)
-
sumApprox
Approximate operation to return the sum within a timeout.- Parameters:
timeout
- (undocumented)confidence
- (undocumented)- Returns:
- (undocumented)
-
sumApprox
Approximate operation to return the sum within a timeout.- Parameters:
timeout
- (undocumented)- Returns:
- (undocumented)
-
histogram
public scala.Tuple2<double[],long[]> histogram(int bucketCount) Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD. For example if the min value is 0 and the max is 100 and there are two buckets the resulting buckets will be [0,50) [50,100]. bucketCount must be at least 1 If the RDD contains infinity, NaN throws an exception If the elements in RDD do not vary (max == min) always returns a single bucket.- Parameters:
bucketCount
- (undocumented)- Returns:
- (undocumented)
-
histogram
public long[] histogram(double[] buckets) Compute a histogram using the provided buckets. The buckets are all open to the left except for the last which is closed e.g. for the array [1,10,20,50] the buckets are [1,10) [10,20) [20,50] e.g 1<=x<10 , 10<=x<20, 20<=x<50 And on the input of 1 and 50 we would have a histogram of 1,0,0- Parameters:
buckets
- (undocumented)- Returns:
- (undocumented)
- Note:
- If your histogram is evenly spaced (e.g. [0, 10, 20, 30]) this can be switched from an O(log n) insertion to O(1) per element. (where n = # buckets) if you set evenBuckets to true. buckets must be sorted and not contain any duplicates. buckets array must be at least two elements All NaN entries are treated the same. If you have a NaN bucket it must be the maximum value of the last position and all NaN entries will be counted in that bucket.
-
histogram
-
setName
Assign a name to this RDD
-