org.apache.spark.streaming

State

sealed abstract class State[S] extends AnyRef

:: Experimental :: Abstract class for getting and updating the state in mapping function used in the mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).

Scala example of using State:

// A mapping function that maintains an integer state and returns a String
def mappingFunction(key: String, value: Option[Int], state: State[Int]): Option[String] = {
  // Check if state exists
  if (state.exists) {
    val existingState = state.get  // Get the existing state
    val shouldRemove = ...         // Decide whether to remove the state
    if (shouldRemove) {
      state.remove()     // Remove the state
    } else {
      val newState = ...
      state.update(newState)    // Set the new state
    }
  } else {
    val initialState = ...
    state.update(initialState)  // Set the initial state
  }
  ... // return something
}

Java example of using State:

// A mapping function that maintains an integer state and returns a String
Function3<String, Optional<Integer>, State<Integer>, String> mappingFunction =
   new Function3<String, Optional<Integer>, State<Integer>, String>() {

     @Override
     public String call(String key, Optional<Integer> value, State<Integer> state) {
       if (state.exists()) {
         int existingState = state.get(); // Get the existing state
         boolean shouldRemove = ...; // Decide whether to remove the state
         if (shouldRemove) {
           state.remove(); // Remove the state
         } else {
           int newState = ...;
           state.update(newState); // Set the new state
         }
       } else {
         int initialState = ...; // Set the initial state
         state.update(initialState);
       }
       // return something
     }
   };
S

Class of the state

Annotations
@Experimental()
Source
State.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. State
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new State()

Abstract Value Members

  1. abstract def exists(): Boolean

    Whether the state already exists

  2. abstract def get(): S

    Get the state if it exists, otherwise it will throw java.util.NoSuchElementException.

    Get the state if it exists, otherwise it will throw java.util.NoSuchElementException. Check with exists() whether the state exists or not before calling get().

    Exceptions thrown
    java.util.NoSuchElementException

    If the state does not exist.

  3. abstract def isTimingOut(): Boolean

    Whether the state is timing out and going to be removed by the system after the current batch.

    Whether the state is timing out and going to be removed by the system after the current batch. This timeout can occur if timeout duration has been specified in the StatSpec and the key has not received any new data for that timeout duration.

  4. abstract def remove(): Unit

    Remove the state if it exists.

    Remove the state if it exists.

    State cannot be updated if it has been already removed (that is, remove() has already been called) or it is going to be removed due to timeout (that is, isTimingOut() is true).

  5. abstract def update(newState: S): Unit

    Update the state with a new value.

    Update the state with a new value.

    State cannot be updated if it has been already removed (that is, remove() has already been called) or it is going to be removed due to timeout (that is, isTimingOut() is true).

    Exceptions thrown
    java.lang.IllegalArgumentException

    If the state has already been removed, or is going to be removed

Concrete Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  11. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  12. final def getOption(): Option[S]

    Get the state as an scala.Option.

    Get the state as an scala.Option. It will be Some(state) if it exists, otherwise None.

    Annotations
    @inline()
  13. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  15. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  16. final def notify(): Unit

    Definition Classes
    AnyRef
  17. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  18. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  19. final def toString(): String

    Definition Classes
    State → AnyRef → Any
    Annotations
    @inline()
  20. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped