org.apache.spark.ml.feature

Class QuantileDiscretizer

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<Bucketizer>

org.apache.spark.ml.feature.QuantileDiscretizer

All Implemented Interfaces:

java.io.Serializable, org.apache.spark.internal.Logging, QuantileDiscretizerBase, Params, HasHandleInvalid, HasInputCol, HasInputCols, HasOutputCol, HasOutputCols, HasRelativeError, DefaultParamsWritable, Identifiable, MLWritable

public final class QuantileDiscretizer
extends Estimator<Bucketizer>
implements QuantileDiscretizerBase, DefaultParamsWritable

QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features. The number of bins can be set using the numBuckets parameter. It is possible that the number of buckets used will be smaller than this value, for example, if there are too few distinct values of the input to create enough distinct quantiles. Since 2.3.0, QuantileDiscretizer can map multiple columns at once by setting the inputCols parameter. If both of the inputCol and inputCols parameters are set, an Exception will be thrown. To specify the number of buckets for each column, the numBucketsArray parameter can be set, or if the number of buckets should be the same across columns, numBuckets can be set as a convenience. Note that in multiple columns case, relative error is applied to all columns.

NaN handling: null and NaN values will be ignored from the column during QuantileDiscretizer fitting. This will produce a Bucketizer model for making predictions. During the transformation, Bucketizer will raise an error when it finds NaN values in the dataset, but the user can also choose to either keep or remove NaN values within the dataset by setting handleInvalid. If the user chooses to keep NaN values, they will be handled specially and placed into their own bucket, for example, if 4 buckets are used, then non-NaN data will be put into buckets[0-3], but NaNs will be counted in a special bucket[4].

Algorithm: The bin ranges are chosen using an approximate algorithm (see the documentation for org.apache.spark.sql.DataFrameStatFunctions.approxQuantile for a detailed description). The precision of the approximation can be controlled with the relativeError parameter. The lower and upper bin bounds will be -Infinity and +Infinity, covering all real values.

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor and Description
QuantileDiscretizer()
QuantileDiscretizer(String uid)

Method Summary

Modifier and Type	Method and Description
QuantileDiscretizer	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
Bucketizer	fit(Dataset<?> dataset) Fits a model to the input data.
Param<String>	handleInvalid() Param for how to handle invalid entries.
Param<String>	inputCol() Param for input column name.
StringArrayParam	inputCols() Param for input column names.
static QuantileDiscretizer	load(String path)
IntParam	numBuckets() Number of buckets (quantiles, or categories) into which data points are grouped.
IntArrayParam	numBucketsArray() Array of number of buckets (quantiles, or categories) into which data points are grouped.
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
Param<String>	outputCol() Param for output column name.
StringArrayParam	outputCols() Param for output column names.
static MLReader<T>	read()
DoubleParam	relativeError() Param for the relative target precision for the approximate quantile algorithm.
QuantileDiscretizer	setHandleInvalid(String value)
QuantileDiscretizer	setInputCol(String value)
QuantileDiscretizer	setInputCols(String[] value)
QuantileDiscretizer	setNumBuckets(int value)
QuantileDiscretizer	setNumBucketsArray(int[] value)
QuantileDiscretizer	setOutputCol(String value)
QuantileDiscretizer	setOutputCols(String[] value)
QuantileDiscretizer	setRelativeError(double value)
StructType	transformSchema(StructType schema) Check transform validity and derive the output schema from the input schema.
String	uid() An immutable unique ID for the object and its derivatives.

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class org.apache.spark.ml.PipelineStage

params

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.feature.QuantileDiscretizerBase

getNumBuckets, getNumBucketsArray

Methods inherited from interface org.apache.spark.ml.param.shared.HasHandleInvalid

getHandleInvalid

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol

getInputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

getOutputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCols

getInputCols

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCols

getOutputCols

Methods inherited from interface org.apache.spark.ml.param.shared.HasRelativeError

getRelativeError

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

write

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.internal.Logging

$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize

Constructor Detail

QuantileDiscretizer

public QuantileDiscretizer(String uid)

QuantileDiscretizer

public QuantileDiscretizer()

Method Detail

load

public static QuantileDiscretizer load(String path)

read

public static MLReader<T> read()

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

numBuckets

public IntParam numBuckets()

Description copied from interface: QuantileDiscretizerBase

Number of buckets (quantiles, or categories) into which data points are grouped. Must be greater than or equal to 2.

See also handleInvalid, which can optionally create an additional bucket for NaN values.

default: 2

Specified by:

numBuckets in interface QuantileDiscretizerBase

Returns:

(undocumented)

numBucketsArray

public IntArrayParam numBucketsArray()

Description copied from interface: QuantileDiscretizerBase

Array of number of buckets (quantiles, or categories) into which data points are grouped. Each value must be greater than or equal to 2

See also handleInvalid, which can optionally create an additional bucket for NaN values.

Specified by:

numBucketsArray in interface QuantileDiscretizerBase

Returns:

(undocumented)

handleInvalid

public Param<String> handleInvalid()

Description copied from interface: QuantileDiscretizerBase

Param for how to handle invalid entries. Options are 'skip' (filter out rows with invalid values), 'error' (throw an error), or 'keep' (keep invalid values in a special additional bucket). Note that in the multiple columns case, the invalid handling is applied to all columns. That said for 'error' it will throw an error if any invalids are found in any column, for 'skip' it will skip rows with any invalids in any columns, etc. Default: "error"

Specified by:

handleInvalid in interface QuantileDiscretizerBase

Specified by:

handleInvalid in interface HasHandleInvalid

Returns:

(undocumented)

relativeError

public final DoubleParam relativeError()

Description copied from interface: HasRelativeError

Param for the relative target precision for the approximate quantile algorithm. Must be in the range [0, 1].

Specified by:

relativeError in interface HasRelativeError

Returns:

(undocumented)

outputCols

public final StringArrayParam outputCols()

Description copied from interface: HasOutputCols

Param for output column names.

Specified by:

outputCols in interface HasOutputCols

Returns:

(undocumented)

inputCols

public final StringArrayParam inputCols()

Description copied from interface: HasInputCols

Param for input column names.

Specified by:

inputCols in interface HasInputCols

Returns:

(undocumented)

outputCol

public final Param<String> outputCol()

Description copied from interface: HasOutputCol

Param for output column name.

Specified by:

outputCol in interface HasOutputCol

Returns:

(undocumented)

inputCol

public final Param<String> inputCol()

Description copied from interface: HasInputCol

Param for input column name.

Specified by:

inputCol in interface HasInputCol

Returns:

(undocumented)

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

setRelativeError

public QuantileDiscretizer setRelativeError(double value)

setNumBuckets

public QuantileDiscretizer setNumBuckets(int value)

setInputCol

public QuantileDiscretizer setInputCol(String value)

setOutputCol

public QuantileDiscretizer setOutputCol(String value)

setHandleInvalid

public QuantileDiscretizer setHandleInvalid(String value)

setNumBucketsArray

public QuantileDiscretizer setNumBucketsArray(int[] value)

setInputCols

public QuantileDiscretizer setInputCols(String[] value)

setOutputCols

public QuantileDiscretizer setOutputCols(String[] value)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

Check transform validity and derive the output schema from the input schema.

We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

fit

public Bucketizer fit(Dataset<?> dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<Bucketizer>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

copy

public QuantileDiscretizer copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<Bucketizer>

Parameters:

extra - (undocumented)

Returns:

(undocumented)