org.apache.spark.sql.streaming

Class StreamingQueryProgress

Object

org.apache.spark.sql.streaming.StreamingQueryProgress

All Implemented Interfaces:

java.io.Serializable

public class StreamingQueryProgress
extends Object
implements scala.Serializable

Information about progress made in the execution of a StreamingQuery during a trigger. Each event relates to processing done for a single trigger of the streaming query. Events are emitted even when no new data is available to be processed.

param: id A unique query id that persists across restarts. See StreamingQuery.id(). param: runId A query id that is unique for every start/restart. See StreamingQuery.runId(). param: name User-specified name of the query, null if not specified. param: timestamp Beginning time of the trigger in ISO8601 format, i.e. UTC timestamps. param: batchId A unique id for the current batch of data being processed. Note that in the case of retries after a failure a given batchId my be executed more than once. Similarly, when there is no data to be processed, the batchId will not be incremented. param: batchDuration The process duration of each batch. param: durationMs The amount of time taken to perform various operations in milliseconds. param: eventTime Statistics of event time seen in this batch. It may contain the following keys:

                   "max" -> "2016-12-05T20:54:20.827Z"  // maximum event time seen in this trigger
                   "min" -> "2016-12-05T20:54:20.827Z"  // minimum event time seen in this trigger
                   "avg" -> "2016-12-05T20:54:20.827Z"  // average event time seen in this trigger
                   "watermark" -> "2016-12-05T20:54:20.827Z"  // watermark used in this trigger
                 

All timestamps are in ISO8601 format, i.e. UTC timestamps. param: stateOperators Information about operators in the query that store state. param: sources detailed statistics on data being read from each of the streaming sources.

Since:

2.1.0

See Also:

Serialized Form

Method Summary

Modifier and Type	Method and Description
long	batchDuration()
long	batchId()
java.util.Map<String,Long>	durationMs()
java.util.Map<String,String>	eventTime()
java.util.UUID	id()
double	inputRowsPerSecond() The aggregate (across all sources) rate of data arriving.
String	json() The compact JSON representation of this progress.
String	name()
long	numInputRows() The aggregate (across all sources) number of records processed in a trigger.
java.util.Map<String,Row>	observedMetrics()
String	prettyJson() The pretty (i.e.
double	processedRowsPerSecond() The aggregate (across all sources) rate at which Spark is processing data.
java.util.UUID	runId()
SinkProgress	sink()
SourceProgress[]	sources()
StateOperatorProgress[]	stateOperators()
String	timestamp()
String	toString()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Method Detail

id

public java.util.UUID id()

runId

public java.util.UUID runId()

name

public String name()

timestamp

public String timestamp()

batchId

public long batchId()

batchDuration

public long batchDuration()

durationMs

public java.util.Map<String,Long> durationMs()

eventTime

public java.util.Map<String,String> eventTime()

stateOperators

public StateOperatorProgress[] stateOperators()

sources

public SourceProgress[] sources()

sink

public SinkProgress sink()

observedMetrics

public java.util.Map<String,Row> observedMetrics()

numInputRows

public long numInputRows()

The aggregate (across all sources) number of records processed in a trigger.

inputRowsPerSecond

public double inputRowsPerSecond()

The aggregate (across all sources) rate of data arriving.

processedRowsPerSecond

public double processedRowsPerSecond()

The aggregate (across all sources) rate at which Spark is processing data.

json

public String json()

The compact JSON representation of this progress.

prettyJson

public String prettyJson()

The pretty (i.e. indented) JSON representation of this progress.

toString

public String toString()

Overrides:

toString in class Object