org.apache.spark.sql.connector.metric

Interface CustomTaskMetric

@Evolving
public interface CustomTaskMetric

A custom task metric. This is a logical representation of a metric reported by data sources at the executor side. During query execution, Spark will collect the task metrics per partition by PartitionReader and update internal metrics based on collected metric values. For streaming query, Spark will collect and combine metrics for a final result per micro batch.

The metrics will be gathered during query execution back to the driver and then combined. How the task metrics are combined is defined by corresponding CustomMetric with same metric name. The final result will be shown up in the data source scan operator in Spark UI.

There are a few special metric names: "bytesWritten" and "recordsWritten". If the data source defines custom metrics with the same names, the metric values will also be updated to corresponding task metrics.

Since:

3.2.0

Method Summary

Modifier and Type	Method and Description
String	name() Returns the name of custom task metric.
long	value() Returns the long value of custom task metric.

Method Detail

name

String name()

Returns the name of custom task metric.

value

long value()

Returns the long value of custom task metric.