Type Members

1.  class ChiSqSelector extends Serializable

   Creates a ChiSquared feature selector.

   Creates a ChiSquared feature selector. The selector supports different selection methods: numTopFeatures, percentile, fpr, fdr, fwe.

   • numTopFeatures chooses a fixed number of top features according to a chi-squared test.
   • percentile is similar but chooses a fraction of all features instead of a fixed number.
   • fpr chooses all features whose p-values are below a threshold, thus controlling the false positive rate of selection.
   • fdr uses the [Benjamini-Hochberg procedure] (https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold.
   • fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection. By default, the selection method is numTopFeatures, with the default number of top features set to 50.

   Annotations

   @Since("1.3.0")

2.  class ChiSqSelectorModel extends VectorTransformer with Saveable

   Chi Squared selector model.

   Chi Squared selector model.

   Annotations

   @Since("1.3.0")

3.  class ElementwiseProduct extends VectorTransformer

   Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.

   Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.

   Annotations

   @Since("1.4.0")

4.  class HashingTF extends Serializable

   Maps a sequence of terms to their term frequencies using the hashing trick.

   Maps a sequence of terms to their term frequencies using the hashing trick.

   Annotations

   @Since("1.1.0")

5.  class IDF extends AnyRef

   Inverse document frequency (IDF).

   Inverse document frequency (IDF). The standard formulation is used: idf = log((m + 1) / (d(t) + 1)), where m is the total number of documents and d(t) is the number of documents that contain term t.

   This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable minDocFreq). For terms that are not in at least minDocFreq documents, the IDF is found as 0, resulting in TF-IDFs of 0. The document frequency is 0 as well for such terms

   Annotations

   @Since("1.1.0")

6.  class IDFModel extends Serializable

   Represents an IDF model that can transform term frequency vectors.

   Represents an IDF model that can transform term frequency vectors.

   Annotations

   @Since("1.1.0")

7.  class Normalizer extends VectorTransformer

   Normalizes samples individually to unit L^p norm

   Normalizes samples individually to unit L^p norm

   For any 1 <= p < Double.PositiveInfinity, normalizes samples using sum(abs(vector).^p)^(1/p) as norm.

   For p = Double.PositiveInfinity, max(abs(vector)) will be used as norm for normalization.

   Annotations

   @Since("1.1.0")

8.  class PCA extends AnyRef

   A feature transformer that projects vectors to a low-dimensional space using PCA.

   A feature transformer that projects vectors to a low-dimensional space using PCA.

   Annotations

   @Since("1.4.0")

9.  class PCAModel extends VectorTransformer

   Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

   Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

   Annotations

   @Since("1.4.0")

10.  class StandardScaler extends Logging

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    The "unit std" is computed using the corrected sample standard deviation (https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation), which is computed as the square root of the unbiased sample variance.

    Annotations

    @Since("1.1.0")

11.  class StandardScalerModel extends VectorTransformer

    Represents a StandardScaler model that can transform vectors.

    Represents a StandardScaler model that can transform vectors.

    Annotations

    @Since("1.1.0")

12.  trait VectorTransformer extends Serializable

    Trait for transformation of a vector

    Trait for transformation of a vector

    Annotations

    @Since("1.1.0")

13.  class Word2Vec extends Serializable with Logging

    Word2Vec creates vector representation of words in a text corpus.

    Word2Vec creates vector representation of words in a text corpus. The algorithm first constructs a vocabulary from the corpus and then learns vector representation of words in the vocabulary. The vector representation can be used as features in natural language processing and machine learning algorithms.

    We used skip-gram model in our implementation and hierarchical softmax method to train the model. The variable names in the implementation matches the original C implementation.

    For original C implementation, see https://code.google.com/p/word2vec/ For research papers, see Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality.

    Annotations

    @Since("1.1.0")

14.  class Word2VecModel extends Serializable with Saveable

    Word2Vec model

    Word2Vec model

    Annotations

    @Since("1.1.0")