Package org.apache.spark.sql.connector.metric

Interface CustomTaskMetric

@Evolving public interface CustomTaskMetric
A custom task metric. This is a logical representation of a metric reported by data sources at the executor side. During query execution, Spark will collect the task metrics per partition by PartitionReader and update internal metrics based on collected metric values. For streaming query, Spark will collect and combine metrics for a final result per micro batch.

The metrics will be gathered during query execution back to the driver and then combined. How the task metrics are combined is defined by corresponding CustomMetric with same metric name. The final result will be shown up in the data source scan operator in Spark UI.

There are a few special metric names: "bytesWritten" and "recordsWritten". If the data source defines custom metrics with the same names, the metric values will also be updated to corresponding task metrics.

Since:
3.2.0

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    name()
    Returns the name of custom task metric.
    long
    value()
    Returns the long value of custom task metric.

  • Method Details

    • name

      String name()
      Returns the name of custom task metric.

    • value

      long value()
      Returns the long value of custom task metric.