Package

org.apache.spark.ml

stat

Permalink

package stat

Visibility
  1. Public
  2. All

Type Members

  1. sealed abstract class SummaryBuilder extends AnyRef

    Permalink

    A builder object that provides summary statistics about a given column.

    A builder object that provides summary statistics about a given column.

    Users should not directly create such builders, but instead use one of the methods in Summarizer.

    Annotations
    @Experimental() @Since( "2.3.0" )

Value Members

  1. object ChiSquareTest

    Permalink

    :: Experimental ::

    :: Experimental ::

    Chi-square hypothesis testing for categorical data.

    See Wikipedia for more information on the Chi-squared test.

    Annotations
    @Experimental() @Since( "2.2.0" )
  2. object Correlation

    Permalink

    API for correlation functions in MLlib, compatible with DataFrames and Datasets.

    API for correlation functions in MLlib, compatible with DataFrames and Datasets.

    The functions in this package generalize the functions in org.apache.spark.sql.Dataset#stat to spark.ml's Vector types.

    Annotations
    @Since( "2.2.0" ) @Experimental()
  3. object KolmogorovSmirnovTest

    Permalink

    :: Experimental ::

    :: Experimental ::

    Conduct the two-sided Kolmogorov Smirnov (KS) test for data sampled from a continuous distribution. By comparing the largest difference between the empirical cumulative distribution of the sample data and the theoretical distribution we can provide a test for the the null hypothesis that the sample data comes from that theoretical distribution. For more information on KS Test:

    Annotations
    @Experimental() @Since( "2.4.0" )
    See also

    Kolmogorov-Smirnov test (Wikipedia)

  4. object Summarizer extends Logging

    Permalink

    Tools for vectorized statistics on MLlib Vectors.

    Tools for vectorized statistics on MLlib Vectors.

    The methods in this package provide various statistics for Vectors contained inside DataFrames.

    This class lets users pick the statistics they would like to extract for a given column. Here is an example in Scala:

    import org.apache.spark.ml.linalg._
    import org.apache.spark.sql.Row
    val dataframe = ... // Some dataframe containing a feature column and a weight column
    val multiStatsDF = dataframe.select(
        Summarizer.metrics("min", "max", "count").summary($"features", $"weight")
    val Row(Row(minVec, maxVec, count)) = multiStatsDF.first()

    If one wants to get a single metric, shortcuts are also available:

    val meanDF = dataframe.select(Summarizer.mean($"features"))
    val Row(meanVec) = meanDF.first()

    Note: Currently, the performance of this interface is about 2x~3x slower than using the RDD interface.

    Annotations
    @Experimental() @Since( "2.3.0" )
  5. package distribution

    Permalink

Members