object Summarizer extends Logging
Tools for vectorized statistics on MLlib Vectors.
The methods in this package provide various statistics for Vectors contained inside DataFrames.
This class lets users pick the statistics they would like to extract for a given column. Here is an example in Scala:
import org.apache.spark.ml.linalg._ import org.apache.spark.sql.Row val dataframe = ... // Some dataframe containing a feature column and a weight column val multiStatsDF = dataframe.select( Summarizer.metrics("min", "max", "count").summary($"features", $"weight") val Row(minVec, maxVec, count) = multiStatsDF.first()
If one wants to get a single metric, shortcuts are also available:
val meanDF = dataframe.select(Summarizer.mean($"features")) val Row(meanVec) = meanDF.first()
Note: Currently, the performance of this interface is about 2x~3x slower than using the RDD interface.
- Annotations
- @Since( "2.3.0" )
- Source
- Summarizer.scala
- Alphabetic
- By Inheritance
- Summarizer
- Logging
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Value Members
-
def
count(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
count(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
max(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
max(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
mean(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
mean(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
metrics(metrics: String*): SummaryBuilder
Given a list of metrics, provides a builder that it turns computes metrics from a column.
Given a list of metrics, provides a builder that it turns computes metrics from a column.
See the documentation of Summarizer for an example.
The following metrics are accepted (case sensitive):
- mean: a vector that contains the coefficient-wise mean.
- sum: a vector that contains the coefficient-wise sum.
- variance: a vector that contains the coefficient-wise variance.
- std: a vector that contains the coefficient-wise standard deviation.
- count: the count of all vectors seen.
- numNonzeros: a vector with the number of non-zeros for each coefficients
- max: the maximum for each coefficient.
- min: the minimum for each coefficient.
- normL2: the Euclidean norm for each coefficient.
- normL1: the L1 norm of each coefficient (sum of the absolute values).
- metrics
metrics that can be provided.
- returns
a builder.
- Annotations
- @Since( "2.3.0" ) @varargs()
- Exceptions thrown
IllegalArgumentException
if one of the metric names is not understood. Note: Currently, the performance of this interface is about 2x~3x slower than using the RDD interface.
-
def
min(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
min(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
normL1(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
normL1(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
normL2(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
normL2(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
numNonZeros(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
numNonZeros(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
std(col: Column): Column
- Annotations
- @Since( "3.0.0" )
-
def
std(col: Column, weightCol: Column): Column
- Annotations
- @Since( "3.0.0" )
-
def
sum(col: Column): Column
- Annotations
- @Since( "3.0.0" )
-
def
sum(col: Column, weightCol: Column): Column
- Annotations
- @Since( "3.0.0" )
-
def
variance(col: Column): Column
- Annotations
- @Since( "2.3.0" )
-
def
variance(col: Column, weightCol: Column): Column
- Annotations
- @Since( "2.3.0" )