org.apache.spark.ml.feature

Class RobustScaler

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<RobustScalerModel>

org.apache.spark.ml.feature.RobustScaler

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.internal.Logging, RobustScalerParams, Params, HasInputCol, HasOutputCol, HasRelativeError, DefaultParamsWritable, Identifiable, MLWritable

public class RobustScaler
extends Estimator<RobustScalerModel>
implements RobustScalerParams, DefaultParamsWritable

Scale features using statistics that are robust to outliers. RobustScaler removes the median and scales the data according to the quantile range. The quantile range is by default IQR (Interquartile Range, quantile range between the 1st quartile = 25th quantile and the 3rd quartile = 75th quantile) but can be configured. Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set. Median and quantile range are then stored to be used on later data using the transform method. Standardization of a dataset is a common requirement for many machine learning estimators. Typically this is done by removing the mean and scaling to unit variance. However, outliers can often influence the sample mean / variance in a negative way. In such cases, the median and the quantile range often give better results. Note that NaN values are ignored in the computation of medians and ranges.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
RobustScaler()
RobustScaler(String uid)

Method Summary

Modifier and Type	Method and Description
RobustScaler	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
RobustScalerModel	fit(Dataset<?> dataset) Fits a model to the input data.
Param<String>	inputCol() Param for input column name.
static RobustScaler	load(String path)
DoubleParam	lower() Lower quantile to calculate quantile range, shared by all features Default: 0.25
Param<String>	outputCol() Param for output column name.
static MLReader<T>	read()
DoubleParam	relativeError() Param for the relative target precision for the approximate quantile algorithm.
RobustScaler	setInputCol(String value)
RobustScaler	setLower(double value)
RobustScaler	setOutputCol(String value)
RobustScaler	setRelativeError(double value)
RobustScaler	setUpper(double value)
RobustScaler	setWithCentering(boolean value)
RobustScaler	setWithScaling(boolean value)
StructType	transformSchema(StructType schema) Check transform validity and derive the output schema from the input schema.
String	uid() An immutable unique ID for the object and its derivatives.
DoubleParam	upper() Upper quantile to calculate quantile range, shared by all features Default: 0.75
BooleanParam	withCentering() Whether to center the data with median before scaling.
BooleanParam	withScaling() Whether to scale the data to quantile range.

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class org.apache.spark.ml.PipelineStage

params

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.feature.RobustScalerParams

getLower, getUpper, getWithCentering, getWithScaling, validateAndTransformSchema

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol

getInputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

getOutputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasRelativeError

getRelativeError

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

write

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

RobustScaler

public RobustScaler(String uid)

RobustScaler

public RobustScaler()

Method Detail

load

public static RobustScaler load(String path)

read

public static MLReader<T> read()

lower

public DoubleParam lower()

Description copied from interface: RobustScalerParams

Lower quantile to calculate quantile range, shared by all features Default: 0.25

Specified by:

lower in interface RobustScalerParams

Returns:

(undocumented)

upper

public DoubleParam upper()

Description copied from interface: RobustScalerParams

Upper quantile to calculate quantile range, shared by all features Default: 0.75

Specified by:

upper in interface RobustScalerParams

Returns:

(undocumented)

withCentering

public BooleanParam withCentering()

Description copied from interface: RobustScalerParams

Whether to center the data with median before scaling. It will build a dense output, so take care when applying to sparse input. Default: false

Specified by:

withCentering in interface RobustScalerParams

Returns:

(undocumented)

withScaling

public BooleanParam withScaling()

Description copied from interface: RobustScalerParams

Whether to scale the data to quantile range. Default: true

Specified by:

withScaling in interface RobustScalerParams

Returns:

(undocumented)

relativeError

public final DoubleParam relativeError()

Description copied from interface: HasRelativeError

Param for the relative target precision for the approximate quantile algorithm. Must be in the range [0, 1].

Specified by:

relativeError in interface HasRelativeError

Returns:

(undocumented)

outputCol

public final Param<String> outputCol()

Description copied from interface: HasOutputCol

Param for output column name.

Specified by:

outputCol in interface HasOutputCol

Returns:

(undocumented)

inputCol

public final Param<String> inputCol()

Description copied from interface: HasInputCol

Param for input column name.

Specified by:

inputCol in interface HasInputCol

Returns:

(undocumented)

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

setInputCol

public RobustScaler setInputCol(String value)

setOutputCol

public RobustScaler setOutputCol(String value)

setLower

public RobustScaler setLower(double value)

setUpper

public RobustScaler setUpper(double value)

setWithCentering

public RobustScaler setWithCentering(boolean value)

setWithScaling

public RobustScaler setWithScaling(boolean value)

setRelativeError

public RobustScaler setRelativeError(double value)

fit

public RobustScalerModel fit(Dataset<?> dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<RobustScalerModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

Check transform validity and derive the output schema from the input schema.

We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy

public RobustScaler copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<RobustScalerModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)