Class

org.apache.spark.sql.streaming

StreamingQueryProgress

Related Doc: package streaming

Permalink

class StreamingQueryProgress extends Serializable

Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

Annotations
@Evolving()
Source
progress.scala
Since

2.1.0

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryProgress
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. val batchId: Long

    Permalink

    A unique id for the current batch of data being processed.

    A unique id for the current batch of data being processed. Note that in the case of retries after a failure a given batchId my be executed more than once. Similarly, when there is no data to be processed, the batchId will not be incremented.

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  7. val durationMs: Map[String, Long]

    Permalink

    The amount of time taken to perform various operations in milliseconds.

  8. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  9. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  10. val eventTime: Map[String, String]

    Permalink

    Statistics of event time seen in this batch.

    Statistics of event time seen in this batch. It may contain the following keys:

    "max" -> "2016-12-05T20:54:20.827Z"  // maximum event time seen in this trigger
    "min" -> "2016-12-05T20:54:20.827Z"  // minimum event time seen in this trigger
    "avg" -> "2016-12-05T20:54:20.827Z"  // average event time seen in this trigger
    "watermark" -> "2016-12-05T20:54:20.827Z"  // watermark used in this trigger

    All timestamps are in ISO8601 format, i.e. UTC timestamps.

  11. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  14. val id: UUID

    Permalink

    An unique query id that persists across restarts.

    An unique query id that persists across restarts. See StreamingQuery.id().

  15. def inputRowsPerSecond: Double

    Permalink

    The aggregate (across all sources) rate of data arriving.

  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  17. def json: String

    Permalink

    The compact JSON representation of this progress.

  18. val name: String

    Permalink

    User-specified name of the query, null if not specified.

  19. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  20. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  21. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  22. def numInputRows: Long

    Permalink

    The aggregate (across all sources) number of records processed in a trigger.

  23. def prettyJson: String

    Permalink

    The pretty (i.e.

    The pretty (i.e. indented) JSON representation of this progress.

  24. def processedRowsPerSecond: Double

    Permalink

    The aggregate (across all sources) rate at which Spark is processing data.

  25. val runId: UUID

    Permalink

    A query id that is unique for every start/restart.

    A query id that is unique for every start/restart. See StreamingQuery.runId().

  26. val sink: SinkProgress

    Permalink
  27. val sources: Array[SourceProgress]

    Permalink

    detailed statistics on data being read from each of the streaming sources.

  28. val stateOperators: Array[StateOperatorProgress]

    Permalink

    Information about operators in the query that store state.

  29. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  30. val timestamp: String

    Permalink

    Beginning time of the trigger in ISO8601 format, i.e.

    Beginning time of the trigger in ISO8601 format, i.e. UTC timestamps.

  31. def toString(): String

    Permalink
    Definition Classes
    StreamingQueryProgress → AnyRef → Any
  32. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  33. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  34. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped