Package org.apache.spark.streaming.dstream

Class MapWithStateDStream<KeyType,ValueType,StateType,MappedType>

Object

org.apache.spark.streaming.dstream.DStream<MappedType>

org.apache.spark.streaming.dstream.MapWithStateDStream<KeyType,ValueType,StateType,MappedType>

Type Parameters:

KeyType - Class of the key

ValueType - Class of the value

StateType - Class of the state data

MappedType - Class of the mapped data

All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging

public abstract class MapWithStateDStream<KeyType,ValueType,StateType,MappedType>
extends DStream<MappedType>

DStream representing the stream of data generated by mapWithState operation on a pair DStream. Additionally, it also gives access to the stream of state snapshots, that is, the state data of all keys after a batch has updated them.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
MapWithStateDStream(StreamingContext ssc, scala.reflect.ClassTag<MappedType> evidence$1)

Method Summary

Modifier and Type	Method	Description
abstract DStream<scala.Tuple2<KeyType,StateType>>	stateSnapshots()	Return a pair DStream where each RDD is the snapshot of the state of all the keys.

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, compute, context, count, countByValue, countByValueAndWindow, countByWindow, dependencies, filter, flatMap, foreachRDD, foreachRDD, glom, map, mapPartitions, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, repartition, saveAsObjectFiles, saveAsTextFiles, slice, slice, slideDuration, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, window, window

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Constructor Details

MapWithStateDStream

public MapWithStateDStream(StreamingContext ssc,
 scala.reflect.ClassTag<MappedType> evidence$1)

Method Details

stateSnapshots

public abstract DStream<scala.Tuple2<KeyType,StateType>> stateSnapshots()

Return a pair DStream where each RDD is the snapshot of the state of all the keys.