Package org.apache.spark.ml.feature

Class ChiSqSelector

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Estimator<T>
org.apache.spark.ml.feature.ChiSqSelector
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, SelectorParams, Params, HasFeaturesCol, HasLabelCol, HasOutputCol, DefaultParamsWritable, Identifiable, MLWritable, scala.Serializable
public final class ChiSqSelector extends Estimator<T>
Deprecated.
use UnivariateFeatureSelector instead. Since 3.1.1.
Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label. The selector supports different selection methods: numTopFeatures, percentile, fpr, fdr, fwe. - numTopFeatures chooses a fixed number of top features according to a chi-squared test. - percentile is similar but chooses a fraction of all features instead of a fixed number. - fpr chooses all features whose p-value are below a threshold, thus controlling the false positive rate of selection. - fdr uses the [Benjamini-Hochberg procedure] (https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold. - fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection. By default, the selection method is numTopFeatures, with the default number of top features set to 50.
See Also:

  • Constructor Details

    • ChiSqSelector

      public ChiSqSelector(String uid)
      Deprecated.

    • ChiSqSelector

      public ChiSqSelector()
      Deprecated.

  • Method Details

    • load

      public static ChiSqSelector load(String path)
      Deprecated.

    • read

      public static MLReader<T> read()
      Deprecated.

    • uid

      public String uid()
      Deprecated.
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Returns:
      (undocumented)

    • setNumTopFeatures

      public ChiSqSelector setNumTopFeatures(int value)
      Deprecated.

    • setPercentile

      public ChiSqSelector setPercentile(double value)
      Deprecated.

    • setFpr

      public ChiSqSelector setFpr(double value)
      Deprecated.

    • setFdr

      public ChiSqSelector setFdr(double value)
      Deprecated.

    • setFwe

      public ChiSqSelector setFwe(double value)
      Deprecated.

    • setSelectorType

      public ChiSqSelector setSelectorType(String value)
      Deprecated.

    • setFeaturesCol

      public ChiSqSelector setFeaturesCol(String value)
      Deprecated.

    • setOutputCol

      public ChiSqSelector setOutputCol(String value)
      Deprecated.

    • setLabelCol

      public ChiSqSelector setLabelCol(String value)
      Deprecated.

    • fit

      public ChiSqSelectorModel fit(Dataset<?> dataset)
      Deprecated.
      Description copied from class: Estimator
      Fits a model to the input data.
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Deprecated.
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)

    • copy

      public ChiSqSelector copy(ParamMap extra)
      Deprecated.
      Description copied from interface: Params
      Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
      Specified by:
      copy in interface Params
      Parameters:
      extra - (undocumented)
      Returns:
      (undocumented)

    • fdr

      public final DoubleParam fdr()
      Description copied from interface: SelectorParams
      The upper bound of the expected false discovery rate. Only applicable when selectorType = "fdr". Default value is 0.05.
      Specified by:
      fdr in interface SelectorParams
      Returns:
      (undocumented)

    • featuresCol

      public final Param<String> featuresCol()
      Description copied from interface: HasFeaturesCol
      Param for features column name.
      Specified by:
      featuresCol in interface HasFeaturesCol
      Returns:
      (undocumented)

    • fpr

      public final DoubleParam fpr()
      Description copied from interface: SelectorParams
      The highest p-value for features to be kept. Only applicable when selectorType = "fpr". Default value is 0.05.
      Specified by:
      fpr in interface SelectorParams
      Returns:
      (undocumented)

    • fwe

      public final DoubleParam fwe()
      Description copied from interface: SelectorParams
      The upper bound of the expected family-wise error rate. Only applicable when selectorType = "fwe". Default value is 0.05.
      Specified by:
      fwe in interface SelectorParams
      Returns:
      (undocumented)

    • labelCol

      public final Param<String> labelCol()
      Description copied from interface: HasLabelCol
      Param for label column name.
      Specified by:
      labelCol in interface HasLabelCol
      Returns:
      (undocumented)

    • numTopFeatures

      public final IntParam numTopFeatures()
      Description copied from interface: SelectorParams
      Number of features that selector will select, ordered by ascending p-value. If the number of features is less than numTopFeatures, then this will select all features. Only applicable when selectorType = "numTopFeatures". The default value of numTopFeatures is 50.

      Specified by:
      numTopFeatures in interface SelectorParams
      Returns:
      (undocumented)

    • outputCol

      public final Param<String> outputCol()
      Description copied from interface: HasOutputCol
      Param for output column name.
      Specified by:
      outputCol in interface HasOutputCol
      Returns:
      (undocumented)

    • percentile

      public final DoubleParam percentile()
      Description copied from interface: SelectorParams
      Percentile of features that selector will select, ordered by ascending p-value. Only applicable when selectorType = "percentile". Default value is 0.1.
      Specified by:
      percentile in interface SelectorParams
      Returns:
      (undocumented)

    • selectorType

      public final Param<String> selectorType()
      Description copied from interface: SelectorParams
      The selector type. Supported options: "numTopFeatures" (default), "percentile", "fpr", "fdr", "fwe"
      Specified by:
      selectorType in interface SelectorParams
      Returns:
      (undocumented)