package feature

Feature transformers

The ml.feature package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting. Most feature transformers are implemented as Transformers, which transform one DataFrame into another, e.g., HashingTF. Some feature transformers are implemented as Estimators, because the transformation requires some aggregated information of the dataset, e.g., document frequencies in IDF. For those feature transformers, calling Estimator!.fit is required to obtain the model first, e.g., IDFModel, in order to apply transformation. The transformation is usually done by appending new columns to the input DataFrame, so all input columns are carried over.

We try to make each transformer minimal, so it becomes flexible to assemble feature transformation pipelines. Pipeline can be used to chain feature transformers, and VectorAssembler can be used to combine multiple feature transformations, for example:


// a DataFrame with three columns: id (integer), text (string), and rating (double).
val df = sqlContext.createDataFrame(Seq(
  (0, "Hi I heard about Spark", 3.0),
  (1, "I wish Java could use case classes", 4.0),
  (2, "Logistic regression models are neat", 4.0)
)).toDF("id", "text", "rating")

// define feature transformers
val tok = new RegexTokenizer()
val sw = new StopWordsRemover()
val tf = new HashingTF()
val idf = new IDF()
val assembler = new VectorAssembler()
  .setInputCols(Array("tf_idf", "rating"))

// assemble and fit the feature transformation pipeline
val pipeline = new Pipeline()
  .setStages(Array(tok, sw, tf, idf, assembler))
val model =

// save transformed features with raw data
  .select("id", "text", "rating", "features")

Some feature transformers implemented in MLlib are inspired by those implemented in scikit-learn. The major difference is that most scikit-learn feature transformers operate eagerly on the entire input dataset, while MLlib's feature transformers operate lazily on individual columns, which is more efficient and flexible to handle large and complex datasets.

See also


Linear Supertypes
AnyRef, Any
  1. Alphabetic
  2. By inheritance
  1. feature
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
  1. Public
  2. All

Type Members

  1. final class Binarizer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: Binarize a column of continuous features given a threshold.

  2. final class Bucketizer extends Model[Bucketizer] with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: Bucketizer maps a column of continuous features to a column of feature buckets.

  3. final class ChiSqSelector extends Estimator[ChiSqSelectorModel] with ChiSqSelectorParams with DefaultParamsWritable

    :: Experimental :: Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.

  4. final class ChiSqSelectorModel extends Model[ChiSqSelectorModel] with ChiSqSelectorParams with MLWritable

    :: Experimental :: Model fitted by ChiSqSelector.

  5. class CountVectorizer extends Estimator[CountVectorizerModel] with CountVectorizerParams with DefaultParamsWritable

    :: Experimental :: Extracts a vocabulary from document collections and generates a CountVectorizerModel.

  6. class CountVectorizerModel extends Model[CountVectorizerModel] with CountVectorizerParams with MLWritable

    :: Experimental :: Converts a text document to a sparse vector of token counts.

  7. class DCT extends UnaryTransformer[Vector, Vector, DCT] with DefaultParamsWritable

    :: Experimental :: A feature transformer that takes the 1D discrete cosine transform of a real vector.

  8. class ElementwiseProduct extends UnaryTransformer[Vector, Vector, ElementwiseProduct]

    :: Experimental :: Outputs the Hadamard product (i.

  9. class HashingTF extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: Maps a sequence of terms to their term frequencies using the hashing trick.

  10. final class IDF extends Estimator[IDFModel] with IDFBase with DefaultParamsWritable

    :: Experimental :: Compute the Inverse Document Frequency (IDF) given a collection of documents.

  11. class IDFModel extends Model[IDFModel] with IDFBase with MLWritable

    :: Experimental :: Model fitted by IDF.

  12. class IndexToString extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: A Transformer that maps a column of indices back to a new column of corresponding string values.

  13. class Interaction extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

    :: Experimental :: Implements the feature interaction transform.

  14. class MinMaxScaler extends Estimator[MinMaxScalerModel] with MinMaxScalerParams with DefaultParamsWritable

    :: Experimental :: Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.

  15. class MinMaxScalerModel extends Model[MinMaxScalerModel] with MinMaxScalerParams with MLWritable

    :: Experimental :: Model fitted by MinMaxScaler.

  16. class NGram extends UnaryTransformer[Seq[String], Seq[String], NGram] with DefaultParamsWritable

    :: Experimental :: A feature transformer that converts the input array of strings into an array of n-grams.

  17. class Normalizer extends UnaryTransformer[Vector, Vector, Normalizer] with DefaultParamsWritable

    :: Experimental :: Normalize a vector to have unit norm using the given p-norm.

  18. class OneHotEncoder extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.

  19. class PCA extends Estimator[PCAModel] with PCAParams with DefaultParamsWritable

    :: Experimental :: PCA trains a model to project vectors to a low-dimensional space using PCA.

  20. class PCAModel extends Model[PCAModel] with PCAParams with MLWritable

    :: Experimental :: Model fitted by PCA.

  21. class PolynomialExpansion extends UnaryTransformer[Vector, Vector, PolynomialExpansion] with DefaultParamsWritable

    :: Experimental :: Perform feature expansion in a polynomial space.

  22. final class QuantileDiscretizer extends Estimator[Bucketizer] with QuantileDiscretizerBase with DefaultParamsWritable

    :: Experimental :: QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features.

  23. class RFormula extends Estimator[RFormulaModel] with RFormulaBase

    :: Experimental :: Implements the transforms required for fitting a dataset against an R model formula.

  24. class RFormulaModel extends Model[RFormulaModel] with RFormulaBase

    :: Experimental :: A fitted RFormula.

  25. class RegexTokenizer extends UnaryTransformer[String, Seq[String], RegexTokenizer] with DefaultParamsWritable

    :: Experimental :: A regex based tokenizer that extracts tokens either by using the provided regex pattern to split the text (default) or repeatedly matching the regex (if gaps is false).

  26. class SQLTransformer extends Transformer with DefaultParamsWritable

    :: Experimental :: Implements the transformations which are defined by SQL statement.

  27. class StandardScaler extends Estimator[StandardScalerModel] with StandardScalerParams with DefaultParamsWritable

    :: Experimental :: Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.

  28. class StandardScalerModel extends Model[StandardScalerModel] with StandardScalerParams with MLWritable

    :: Experimental :: Model fitted by StandardScaler.

  29. class StopWordsRemover extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: A feature transformer that filters out stop words from input.

  30. class StringIndexer extends Estimator[StringIndexerModel] with StringIndexerBase with DefaultParamsWritable

    :: Experimental :: A label indexer that maps a string column of labels to an ML column of label indices.

  31. class StringIndexerModel extends Model[StringIndexerModel] with StringIndexerBase with MLWritable

    :: Experimental :: Model fitted by StringIndexer.

  32. class Tokenizer extends UnaryTransformer[String, Seq[String], Tokenizer] with DefaultParamsWritable

    :: Experimental :: A tokenizer that converts the input string to lowercase and then splits it by white spaces.

  33. class VectorAssembler extends Transformer with HasInputCols with HasOutputCol with DefaultParamsWritable

    :: Experimental :: A feature transformer that merges multiple columns into a vector column.

  34. class VectorIndexer extends Estimator[VectorIndexerModel] with VectorIndexerParams with DefaultParamsWritable

    :: Experimental :: Class for indexing categorical feature columns in a dataset of Vector.

  35. class VectorIndexerModel extends Model[VectorIndexerModel] with VectorIndexerParams with MLWritable

    :: Experimental :: Transform categorical features to use 0-based indices instead of their original values.

  36. final class VectorSlicer extends Transformer with HasInputCol with HasOutputCol with DefaultParamsWritable

    :: Experimental :: This class takes a feature vector and outputs a new feature vector with a subarray of the original features.

  37. final class Word2Vec extends Estimator[Word2VecModel] with Word2VecBase with DefaultParamsWritable

    :: Experimental :: Word2Vec trains a model of Map(String, Vector), i.

  38. class Word2VecModel extends Model[Word2VecModel] with Word2VecBase with MLWritable

    :: Experimental :: Model fitted by Word2Vec.

Value Members

  1. object Binarizer extends DefaultParamsReadable[Binarizer] with Serializable

    @Since( "1.6.0" )
  2. object Bucketizer extends DefaultParamsReadable[Bucketizer] with Serializable

  3. object ChiSqSelector extends DefaultParamsReadable[ChiSqSelector] with Serializable

    @Since( "1.6.0" )
  4. object ChiSqSelectorModel extends MLReadable[ChiSqSelectorModel] with Serializable

    @Since( "1.6.0" )
  5. object CountVectorizer extends DefaultParamsReadable[CountVectorizer] with Serializable

    @Since( "1.6.0" )
  6. object CountVectorizerModel extends MLReadable[CountVectorizerModel] with Serializable

    @Since( "1.6.0" )
  7. object DCT extends DefaultParamsReadable[DCT] with Serializable

    @Since( "1.6.0" )
  8. object HashingTF extends DefaultParamsReadable[HashingTF] with Serializable

    @Since( "1.6.0" )
  9. object IDF extends DefaultParamsReadable[IDF] with Serializable

    @Since( "1.6.0" )
  10. object IDFModel extends MLReadable[IDFModel] with Serializable

    @Since( "1.6.0" )
  11. object IndexToString extends DefaultParamsReadable[IndexToString] with Serializable

    @Since( "1.6.0" )
  12. object Interaction extends DefaultParamsReadable[Interaction] with Serializable

    @Since( "1.6.0" )
  13. object MinMaxScaler extends DefaultParamsReadable[MinMaxScaler] with Serializable

    @Since( "1.6.0" )
  14. object MinMaxScalerModel extends MLReadable[MinMaxScalerModel] with Serializable

    @Since( "1.6.0" )
  15. object NGram extends DefaultParamsReadable[NGram] with Serializable

    @Since( "1.6.0" )
  16. object Normalizer extends DefaultParamsReadable[Normalizer] with Serializable

    @Since( "1.6.0" )
  17. object OneHotEncoder extends DefaultParamsReadable[OneHotEncoder] with Serializable

    @Since( "1.6.0" )
  18. object PCA extends DefaultParamsReadable[PCA] with Serializable

    @Since( "1.6.0" )
  19. object PCAModel extends MLReadable[PCAModel] with Serializable

    @Since( "1.6.0" )
  20. object PolynomialExpansion extends DefaultParamsReadable[PolynomialExpansion] with Serializable

    The expansion is done via recursion.

  21. object QuantileDiscretizer extends DefaultParamsReadable[QuantileDiscretizer] with Logging with Serializable

    @Since( "1.6.0" )
  22. object RegexTokenizer extends DefaultParamsReadable[RegexTokenizer] with Serializable

    @Since( "1.6.0" )
  23. object SQLTransformer extends DefaultParamsReadable[SQLTransformer] with Serializable

    @Since( "1.6.0" )
  24. object StandardScaler extends DefaultParamsReadable[StandardScaler] with Serializable

    @Since( "1.6.0" )
  25. object StandardScalerModel extends MLReadable[StandardScalerModel] with Serializable

    @Since( "1.6.0" )
  26. object StopWordsRemover extends DefaultParamsReadable[StopWordsRemover] with Serializable

    @Since( "1.6.0" )
  27. object StringIndexer extends DefaultParamsReadable[StringIndexer] with Serializable

    @Since( "1.6.0" )
  28. object StringIndexerModel extends MLReadable[StringIndexerModel] with Serializable

    @Since( "1.6.0" )
  29. object Tokenizer extends DefaultParamsReadable[Tokenizer] with Serializable

    @Since( "1.6.0" )
  30. object VectorAssembler extends DefaultParamsReadable[VectorAssembler] with Serializable

    @Since( "1.6.0" )
  31. object VectorIndexer extends DefaultParamsReadable[VectorIndexer] with Serializable

    @Since( "1.6.0" )
  32. object VectorIndexerModel extends MLReadable[VectorIndexerModel] with Serializable

    @Since( "1.6.0" )
  33. object VectorSlicer extends DefaultParamsReadable[VectorSlicer] with Serializable

    @Since( "1.6.0" )
  34. object Word2Vec extends DefaultParamsReadable[Word2Vec] with Serializable

    @Since( "1.6.0" )
  35. object Word2VecModel extends MLReadable[Word2VecModel] with Serializable

    @Since( "1.6.0" )

Inherited from AnyRef

Inherited from Any