Type Members

1.  abstract class CustomAvgMetric extends CustomMetric

   Built-in CustomMetric that computes average of metric values.

   Built-in CustomMetric that computes average of metric values. Note that please extend this class and override name and description to create your custom metric for real usage.

   Annotations

   @Evolving()

   Since

   3.2.0

2.  trait CustomMetric extends AnyRef

   A custom metric.

   A custom metric. Data source can define supported custom metrics using this interface. During query execution, Spark will collect the task metrics using CustomTaskMetric and combine the metrics at the driver side. How to combine task metrics is defined by the metric class with the same metric name.

   When Spark needs to aggregate task metrics, it will internally construct the instance of custom metric class defined in data source by using reflection. Spark requires the class implementing this interface to have a 0-arg constructor.

   Annotations

   @Evolving()

   Since

   3.2.0

3.  abstract class CustomSumMetric extends CustomMetric

   Built-in CustomMetric that sums up metric values.

   Built-in CustomMetric that sums up metric values. Note that please extend this class and override name and description to create your custom metric for real usage.

   Annotations

   @Evolving()

   Since

   3.2.0

4.  trait CustomTaskMetric extends AnyRef

   A custom task metric.

   A custom task metric. This is a logical representation of a metric reported by data sources at the executor side. During query execution, Spark will collect the task metrics per partition by PartitionReader and update internal metrics based on collected metric values. For streaming query, Spark will collect and combine metrics for a final result per micro batch.

   The metrics will be gathered during query execution back to the driver and then combined. How the task metrics are combined is defined by corresponding CustomMetric with same metric name. The final result will be shown up in the data source scan operator in Spark UI.

   There are a few special metric names: "bytesWritten" and "recordsWritten". If the data source defines custom metrics with the same names, the metric values will also be updated to corresponding task metrics.

   Annotations

   @Evolving()

   Since

   3.2.0