Class JavaDStream<T>
Object
org.apache.spark.streaming.api.java.JavaDStream<T>
- All Implemented Interfaces:
Serializable
,JavaDStreamLike<T,
,JavaDStream<T>, JavaRDD<T>> scala.Serializable
- Direct Known Subclasses:
JavaInputDStream
,JavaMapWithStateDStream
A Java-friendly interface to
DStream
, the basic
abstraction in Spark Streaming that represents a continuous stream of data.
DStreams can either be created from live data (such as, data from TCP sockets, Kafka,
etc.) or it can be generated by transforming existing DStreams using operations such as map
,
window
. For operations applicable to key-value pair DStreams, see
JavaPairDStream
.- See Also:
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptioncache()
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)scala.reflect.ClassTag<T>
classTag()
Generate an RDD for the given durationdstream()
Return a new DStream containing only the elements that satisfy a predicate.static <T> JavaDStream<T>
fromDStream
(DStream<T> dstream, scala.reflect.ClassTag<T> evidence$1) Convert a scalaDStream
to a Java-friendlyJavaDStream
.persist()
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)persist
(StorageLevel storageLevel) Persist the RDDs of this DStream with the given storage levelrepartition
(int numPartitions) Return a new DStream with an increased or decreased level of parallelism.union
(JavaDStream<T> that) Return a new DStream by unifying data of another DStream with this DStream.Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.spark.streaming.api.java.JavaDStreamLike
checkpoint, context, count, countByValue, countByValue, countByValueAndWindow, countByValueAndWindow, countByWindow, flatMap, flatMapToPair, foreachRDD, foreachRDD, glom, map, mapPartitions, mapPartitionsToPair, mapToPair, print, print, reduce, reduceByWindow, reduceByWindow, scalaIntToJavaLong, slice, transform, transform, transformToPair, transformToPair, transformWith, transformWith, transformWithToPair, transformWithToPair
-
Constructor Details
-
JavaDStream
-
-
Method Details
-
fromDStream
public static <T> JavaDStream<T> fromDStream(DStream<T> dstream, scala.reflect.ClassTag<T> evidence$1) Convert a scalaDStream
to a Java-friendlyJavaDStream
.- Parameters:
dstream
- (undocumented)evidence$1
- (undocumented)- Returns:
- (undocumented)
-
dstream
-
classTag
-
wrapRDD
-
filter
Return a new DStream containing only the elements that satisfy a predicate. -
cache
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER) -
persist
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER) -
persist
Persist the RDDs of this DStream with the given storage level -
compute
Generate an RDD for the given duration -
window
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream. The new DStream generates RDDs with the same interval as this DStream.- Parameters:
windowDuration
- width of the window; must be a multiple of this DStream's interval.- Returns:
- (undocumented)
-
window
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.- Parameters:
windowDuration
- width of the window; must be a multiple of this DStream's batching intervalslideDuration
- sliding interval of the window (i.e., the interval after which the new DStream will generate RDDs); must be a multiple of this DStream's batching interval- Returns:
- (undocumented)
-
union
Return a new DStream by unifying data of another DStream with this DStream.- Parameters:
that
- Another DStream having the same interval (i.e., slideDuration) as this DStream.- Returns:
- (undocumented)
-
repartition
Return a new DStream with an increased or decreased level of parallelism. Each RDD in the returned DStream has exactly numPartitions partitions.- Parameters:
numPartitions
- (undocumented)- Returns:
- (undocumented)
-