Package org.apache.spark.ml.feature

Class UnivariateFeatureSelector

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Estimator<UnivariateFeatureSelectorModel>
org.apache.spark.ml.feature.UnivariateFeatureSelector
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, org.apache.spark.ml.feature.UnivariateFeatureSelectorParams, Params, HasFeaturesCol, HasLabelCol, HasOutputCol, DefaultParamsWritable, Identifiable, MLWritable
public final class UnivariateFeatureSelector extends Estimator<UnivariateFeatureSelectorModel> implements org.apache.spark.ml.feature.UnivariateFeatureSelectorParams, DefaultParamsWritable
Feature selector based on univariate statistical tests against labels. Currently, Spark supports three Univariate Feature Selectors: chi-squared, ANOVA F-test and F-value. User can choose Univariate Feature Selector by setting featureType and labelType, and Spark will pick the score function based on the specified featureType and labelType.

The following combination of featureType and labelType are supported: - featureType categorical and labelType categorical: Spark uses chi-squared, i.e. chi2 in sklearn. - featureType continuous and labelType categorical: Spark uses ANOVA F-test, i.e. f_classif in sklearn. - featureType continuous and labelType continuous: Spark uses F-value, i.e. f_regression in sklearn.

The UnivariateFeatureSelector supports different selection modes: numTopFeatures, percentile, fpr, fdr, fwe. - numTopFeatures chooses a fixed number of top features according to a hypothesis. - percentile is similar but chooses a fraction of all features instead of a fixed number. - fpr chooses all features whose p-value are below a threshold, thus controlling the false positive rate of selection. - fdr uses the Benjamini-Hochberg procedure to choose all features whose false discovery rate is below a threshold. - fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection.

By default, the selection mode is numTopFeatures.

See Also:

  • Constructor Details

    • UnivariateFeatureSelector

      public UnivariateFeatureSelector(String uid)

    • UnivariateFeatureSelector

      public UnivariateFeatureSelector()

  • Method Details

    • load

      public static UnivariateFeatureSelector load(String path)

    • read

      public static MLReader<T> read()

    • featureType

      public final Param<String> featureType()
      Specified by:
      featureType in interface org.apache.spark.ml.feature.UnivariateFeatureSelectorParams

    • labelType

      public final Param<String> labelType()
      Specified by:
      labelType in interface org.apache.spark.ml.feature.UnivariateFeatureSelectorParams

    • selectionMode

      public final Param<String> selectionMode()
      Specified by:
      selectionMode in interface org.apache.spark.ml.feature.UnivariateFeatureSelectorParams

    • selectionThreshold

      public final DoubleParam selectionThreshold()
      Specified by:
      selectionThreshold in interface org.apache.spark.ml.feature.UnivariateFeatureSelectorParams

    • outputCol

      public final Param<String> outputCol()
      Description copied from interface: HasOutputCol
      Param for output column name.
      Specified by:
      outputCol in interface HasOutputCol
      Returns:
      (undocumented)

    • labelCol

      public final Param<String> labelCol()
      Description copied from interface: HasLabelCol
      Param for label column name.
      Specified by:
      labelCol in interface HasLabelCol
      Returns:
      (undocumented)

    • featuresCol

      public final Param<String> featuresCol()
      Description copied from interface: HasFeaturesCol
      Param for features column name.
      Specified by:
      featuresCol in interface HasFeaturesCol
      Returns:
      (undocumented)

    • uid

      public String uid()
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Specified by:
      uid in interface Identifiable
      Returns:
      (undocumented)

    • setSelectionMode

      public UnivariateFeatureSelector setSelectionMode(String value)

    • setSelectionThreshold

      public UnivariateFeatureSelector setSelectionThreshold(double value)

    • setFeatureType

      public UnivariateFeatureSelector setFeatureType(String value)

    • setLabelType

      public UnivariateFeatureSelector setLabelType(String value)

    • setFeaturesCol

      public UnivariateFeatureSelector setFeaturesCol(String value)

    • setOutputCol

      public UnivariateFeatureSelector setOutputCol(String value)

    • setLabelCol

      public UnivariateFeatureSelector setLabelCol(String value)

    • fit

      public UnivariateFeatureSelectorModel fit(Dataset<?> dataset)
      Description copied from class: Estimator
      Fits a model to the input data.
      Specified by:
      fit in class Estimator<UnivariateFeatureSelectorModel>
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Specified by:
      transformSchema in class PipelineStage
      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)

    • copy

      public UnivariateFeatureSelector copy(ParamMap extra)
      Description copied from interface: Params
      Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
      Specified by:
      copy in interface Params
      Specified by:
      copy in class Estimator<UnivariateFeatureSelectorModel>
      Parameters:
      extra - (undocumented)
      Returns:
      (undocumented)