trait CustomTaskMetric extends AnyRef
A custom task metric. This is a logical representation of a metric reported by data sources
at the executor side. During query execution, Spark will collect the task metrics per partition
by PartitionReader and update internal metrics based on collected metric values.
For streaming query, Spark will collect and combine metrics for a final result per micro batch.
The metrics will be gathered during query execution back to the driver and then combined. How
the task metrics are combined is defined by corresponding CustomMetric with same metric
name. The final result will be shown up in the data source scan operator in Spark UI.
There are a few special metric names: "bytesWritten" and "recordsWritten". If the data source defines custom metrics with the same names, the metric values will also be updated to corresponding task metrics.
- Annotations
- @Evolving()
- Source
- CustomTaskMetric.java
- Since
3.2.0
- Alphabetic
- By Inheritance
- CustomTaskMetric
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def mergeWith(other: CustomTaskMetric): CustomTaskMetric
Merges this metric with another metric of the same name, returning a new
CustomTaskMetricthat represents the combined value.Merges this metric with another metric of the same name, returning a new
CustomTaskMetricthat represents the combined value. This is called when a task reads multiple partitions concurrently (e.g., k-way merge coalescing) to produce a single task-level value before reporting to the driver.The default implementation returns a new metric whose value is the sum of the two values, which is correct for count-type metrics. Data sources with non-additive metrics (e.g., max, average, last-value) must override this method to provide correct merge semantics.
- other
another metric with the same name to merge with
- returns
a new metric representing the merged value
- Since
4.2.0
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)