public final class UnivariateFeatureSelector extends Estimator<UnivariateFeatureSelectorModel> implements UnivariateFeatureSelectorParams, DefaultParamsWritable
featureType and labelType,
and Spark will pick the score function based on the specified featureType and labelType.
The following combination of featureType and labelType are supported:
- featureType categorical and labelType categorical: Spark uses chi-squared,
i.e. chi2 in sklearn.
- featureType continuous and labelType categorical: Spark uses ANOVA F-test,
i.e. f_classif in sklearn.
- featureType continuous and labelType continuous: Spark uses F-value,
i.e. f_regression in sklearn.
The UnivariateFeatureSelector supports different selection modes: numTopFeatures,
percentile, fpr, fdr, fwe.
- numTopFeatures chooses a fixed number of top features according to a hypothesis.
- percentile is similar but chooses a fraction of all features instead of a fixed number.
- fpr chooses all features whose p-value are below a threshold, thus controlling the false
positive rate of selection.
- fdr uses the
Benjamini-Hochberg procedure
to choose all features whose false discovery rate is below a threshold.
- fwe chooses all features whose p-values are below a threshold. The threshold is scaled by
1/numFeatures, thus controlling the family-wise error rate of selection.
By default, the selection mode is numTopFeatures.
| Constructor and Description |
|---|
UnivariateFeatureSelector() |
UnivariateFeatureSelector(String uid) |
| Modifier and Type | Method and Description |
|---|---|
UnivariateFeatureSelector |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
Param<String> |
featuresCol()
Param for features column name.
|
Param<String> |
featureType()
The feature type.
|
UnivariateFeatureSelectorModel |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
Param<String> |
labelCol()
Param for label column name.
|
Param<String> |
labelType()
The label type.
|
static UnivariateFeatureSelector |
load(String path) |
Param<String> |
outputCol()
Param for output column name.
|
static MLReader<T> |
read() |
Param<String> |
selectionMode()
The selection mode.
|
DoubleParam |
selectionThreshold()
The upper bound of the features that selector will select.
|
UnivariateFeatureSelector |
setFeaturesCol(String value) |
UnivariateFeatureSelector |
setFeatureType(String value) |
UnivariateFeatureSelector |
setLabelCol(String value) |
UnivariateFeatureSelector |
setLabelType(String value) |
UnivariateFeatureSelector |
setOutputCol(String value) |
UnivariateFeatureSelector |
setSelectionMode(String value) |
UnivariateFeatureSelector |
setSelectionThreshold(double value) |
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
paramsequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetFeatureType, getLabelType, getSelectionMode, getSelectionThresholdgetFeaturesColgetLabelColgetOutputColclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringwritesave$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitializepublic UnivariateFeatureSelector(String uid)
public UnivariateFeatureSelector()
public static UnivariateFeatureSelector load(String path)
public static MLReader<T> read()
public final Param<String> featureType()
UnivariateFeatureSelectorParamsfeatureType in interface UnivariateFeatureSelectorParamspublic final Param<String> labelType()
UnivariateFeatureSelectorParamslabelType in interface UnivariateFeatureSelectorParamspublic final Param<String> selectionMode()
UnivariateFeatureSelectorParamsselectionMode in interface UnivariateFeatureSelectorParamspublic final DoubleParam selectionThreshold()
UnivariateFeatureSelectorParamsselectionThreshold in interface UnivariateFeatureSelectorParamspublic final Param<String> outputCol()
HasOutputColoutputCol in interface HasOutputColpublic final Param<String> labelCol()
HasLabelCollabelCol in interface HasLabelColpublic final Param<String> featuresCol()
HasFeaturesColfeaturesCol in interface HasFeaturesColpublic String uid()
Identifiableuid in interface Identifiablepublic UnivariateFeatureSelector setSelectionMode(String value)
public UnivariateFeatureSelector setSelectionThreshold(double value)
public UnivariateFeatureSelector setFeatureType(String value)
public UnivariateFeatureSelector setLabelType(String value)
public UnivariateFeatureSelector setFeaturesCol(String value)
public UnivariateFeatureSelector setOutputCol(String value)
public UnivariateFeatureSelector setLabelCol(String value)
public UnivariateFeatureSelectorModel fit(Dataset<?> dataset)
Estimatorfit in class Estimator<UnivariateFeatureSelectorModel>dataset - (undocumented)public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)public UnivariateFeatureSelector copy(ParamMap extra)
ParamsdefaultCopy().copy in interface Paramscopy in class Estimator<UnivariateFeatureSelectorModel>extra - (undocumented)