Package org.apache.spark.ml.feature

Class UnivariateFeatureSelector

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<UnivariateFeatureSelectorModel>

org.apache.spark.ml.feature.UnivariateFeatureSelector

All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging, UnivariateFeatureSelectorParams, Params, HasFeaturesCol, HasLabelCol, HasOutputCol, DefaultParamsWritable, Identifiable, MLWritable, scala.Serializable

public final class UnivariateFeatureSelector
extends Estimator<UnivariateFeatureSelectorModel>
implements UnivariateFeatureSelectorParams, DefaultParamsWritable

Feature selector based on univariate statistical tests against labels. Currently, Spark supports three Univariate Feature Selectors: chi-squared, ANOVA F-test and F-value. User can choose Univariate Feature Selector by setting featureType and labelType, and Spark will pick the score function based on the specified featureType and labelType.

The following combination of featureType and labelType are supported: - featureType categorical and labelType categorical: Spark uses chi-squared, i.e. chi2 in sklearn. - featureType continuous and labelType categorical: Spark uses ANOVA F-test, i.e. f_classif in sklearn. - featureType continuous and labelType continuous: Spark uses F-value, i.e. f_regression in sklearn.

The UnivariateFeatureSelector supports different selection modes: numTopFeatures, percentile, fpr, fdr, fwe. - numTopFeatures chooses a fixed number of top features according to a hypothesis. - percentile is similar but chooses a fraction of all features instead of a fixed number. - fpr chooses all features whose p-value are below a threshold, thus controlling the false positive rate of selection. - fdr uses the Benjamini-Hochberg procedure to choose all features whose false discovery rate is below a threshold. - fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection.

By default, the selection mode is numTopFeatures.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
UnivariateFeatureSelector()
UnivariateFeatureSelector(String uid)

Method Summary

Modifier and Type	Method	Description
UnivariateFeatureSelector	copy(ParamMap extra)	Creates a copy of this instance with the same UID and some extra params.
final Param<String>	featuresCol()	Param for features column name.
final Param<String>	featureType()	The feature type.
UnivariateFeatureSelectorModel	fit(Dataset<?> dataset)	Fits a model to the input data.
final Param<String>	labelCol()	Param for label column name.
final Param<String>	labelType()	The label type.
static UnivariateFeatureSelector	load(String path)
final Param<String>	outputCol()	Param for output column name.
static MLReader<T>	read()
final Param<String>	selectionMode()	The selection mode.
final DoubleParam	selectionThreshold()	The upper bound of the features that selector will select.
UnivariateFeatureSelector	setFeaturesCol(String value)
UnivariateFeatureSelector	setFeatureType(String value)
UnivariateFeatureSelector	setLabelCol(String value)
UnivariateFeatureSelector	setLabelType(String value)
UnivariateFeatureSelector	setOutputCol(String value)
UnivariateFeatureSelector	setSelectionMode(String value)
UnivariateFeatureSelector	setSelectionThreshold(double value)
StructType	transformSchema(StructType schema)	Check transform validity and derive the output schema from the input schema.
String	uid()	An immutable unique ID for the object and its derivatives.

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class org.apache.spark.ml.PipelineStage

params

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

write

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol

getFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasLabelCol

getLabelCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

getOutputCol

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.feature.UnivariateFeatureSelectorParams

getFeatureType, getLabelType, getSelectionMode, getSelectionThreshold

Constructor Details

UnivariateFeatureSelector

public UnivariateFeatureSelector(String uid)

UnivariateFeatureSelector

public UnivariateFeatureSelector()

Method Details

load

public static UnivariateFeatureSelector load(String path)

read

public static MLReader<T> read()

featureType

public final Param<String> featureType()

Description copied from interface: UnivariateFeatureSelectorParams

The feature type. Supported options: "categorical", "continuous"

Specified by:

featureType in interface UnivariateFeatureSelectorParams

Returns:

(undocumented)

labelType

public final Param<String> labelType()

Description copied from interface: UnivariateFeatureSelectorParams

The label type. Supported options: "categorical", "continuous"

Specified by:

labelType in interface UnivariateFeatureSelectorParams

Returns:

(undocumented)

selectionMode

public final Param<String> selectionMode()

Description copied from interface: UnivariateFeatureSelectorParams

The selection mode. Supported options: "numTopFeatures" (default), "percentile", "fpr", "fdr", "fwe"

Specified by:

selectionMode in interface UnivariateFeatureSelectorParams

Returns:

(undocumented)

selectionThreshold

public final DoubleParam selectionThreshold()

Description copied from interface: UnivariateFeatureSelectorParams

The upper bound of the features that selector will select.

Specified by:

selectionThreshold in interface UnivariateFeatureSelectorParams

Returns:

(undocumented)

outputCol

public final Param<String> outputCol()

Description copied from interface: HasOutputCol

Param for output column name.

Specified by:

outputCol in interface HasOutputCol

Returns:

(undocumented)

labelCol

public final Param<String> labelCol()

Description copied from interface: HasLabelCol

Param for label column name.

Specified by:

labelCol in interface HasLabelCol

Returns:

(undocumented)

featuresCol

public final Param<String> featuresCol()

Description copied from interface: HasFeaturesCol

Param for features column name.

Specified by:

featuresCol in interface HasFeaturesCol

Returns:

(undocumented)

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

setSelectionMode

public UnivariateFeatureSelector setSelectionMode(String value)

setSelectionThreshold

public UnivariateFeatureSelector setSelectionThreshold(double value)

setFeatureType

public UnivariateFeatureSelector setFeatureType(String value)

setLabelType

public UnivariateFeatureSelector setLabelType(String value)

setFeaturesCol

public UnivariateFeatureSelector setFeaturesCol(String value)

setOutputCol

public UnivariateFeatureSelector setOutputCol(String value)

setLabelCol

public UnivariateFeatureSelector setLabelCol(String value)

fit

public UnivariateFeatureSelectorModel fit(Dataset<?> dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<UnivariateFeatureSelectorModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

Check transform validity and derive the output schema from the input schema.

We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy

public UnivariateFeatureSelector copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<UnivariateFeatureSelectorModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)