Packages

c

org.apache.spark.sql.streaming

StreamingQueryProgress

class StreamingQueryProgress extends Serializable

Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

Annotations
@Evolving()
Source
progress.scala
Since

2.1.0

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. StreamingQueryProgress
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. val batchDuration: Long
  2. val batchId: Long
  3. val durationMs: Map[String, Long]
  4. val eventTime: Map[String, String]
  5. val id: UUID
  6. def inputRowsPerSecond: Double

    The aggregate (across all sources) rate of data arriving.

  7. def json: String

    The compact JSON representation of this progress.

  8. val name: String
  9. def numInputRows: Long

    The aggregate (across all sources) number of records processed in a trigger.

  10. val observedMetrics: Map[String, Row]
  11. def prettyJson: String

    The pretty (i.e.

    The pretty (i.e. indented) JSON representation of this progress.

  12. def processedRowsPerSecond: Double

    The aggregate (across all sources) rate at which Spark is processing data.

  13. val runId: UUID
  14. val sink: SinkProgress
  15. val sources: Array[SourceProgress]
  16. val stateOperators: Array[StateOperatorProgress]
  17. val timestamp: String
  18. def toString(): String
    Definition Classes
    StreamingQueryProgress → AnyRef → Any