Packages

package feature

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class ChiSqSelector extends Serializable

    Creates a ChiSquared feature selector.

    Creates a ChiSquared feature selector. The selector supports different selection methods: numTopFeatures, percentile, fpr, fdr, fwe.

    • numTopFeatures chooses a fixed number of top features according to a chi-squared test.
    • percentile is similar but chooses a fraction of all features instead of a fixed number.
    • fpr chooses all features whose p-values are below a threshold, thus controlling the false positive rate of selection.
    • fdr uses the [Benjamini-Hochberg procedure] (https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold.
    • fwe chooses all features whose p-values are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the family-wise error rate of selection. By default, the selection method is numTopFeatures, with the default number of top features set to 50.
    Annotations
    @Since( "1.3.0" )
  2. class ChiSqSelectorModel extends VectorTransformer with Saveable

    Chi Squared selector model.

    Chi Squared selector model.

    Annotations
    @Since( "1.3.0" )
  3. class ElementwiseProduct extends VectorTransformer

    Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.

    Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.

    Annotations
    @Since( "1.4.0" )
  4. class HashingTF extends Serializable

    Maps a sequence of terms to their term frequencies using the hashing trick.

    Maps a sequence of terms to their term frequencies using the hashing trick.

    Annotations
    @Since( "1.1.0" )
  5. class IDF extends AnyRef

    Inverse document frequency (IDF).

    Inverse document frequency (IDF). The standard formulation is used: idf = log((m + 1) / (d(t) + 1)), where m is the total number of documents and d(t) is the number of documents that contain term t.

    This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable minDocFreq). For terms that are not in at least minDocFreq documents, the IDF is found as 0, resulting in TF-IDFs of 0. The document frequency is 0 as well for such terms

    Annotations
    @Since( "1.1.0" )
  6. class IDFModel extends Serializable

    Represents an IDF model that can transform term frequency vectors.

    Represents an IDF model that can transform term frequency vectors.

    Annotations
    @Since( "1.1.0" )
  7. class Normalizer extends VectorTransformer

    Normalizes samples individually to unit Lp norm

    Normalizes samples individually to unit Lp norm

    For any 1 <= p < Double.PositiveInfinity, normalizes samples using sum(abs(vector).p)(1/p) as norm.

    For p = Double.PositiveInfinity, max(abs(vector)) will be used as norm for normalization.

    Annotations
    @Since( "1.1.0" )
  8. class PCA extends AnyRef

    A feature transformer that projects vectors to a low-dimensional space using PCA.

    A feature transformer that projects vectors to a low-dimensional space using PCA.

    Annotations
    @Since( "1.4.0" )
  9. class PCAModel extends VectorTransformer

    Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

    Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

    Annotations
    @Since( "1.4.0" )
  10. class StandardScaler extends Logging

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    The "unit std" is computed using the corrected sample standard deviation (https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation), which is computed as the square root of the unbiased sample variance.

    Annotations
    @Since( "1.1.0" )
  11. class StandardScalerModel extends VectorTransformer

    Represents a StandardScaler model that can transform vectors.

    Represents a StandardScaler model that can transform vectors.

    Annotations
    @Since( "1.1.0" )
  12. trait VectorTransformer extends Serializable

    Trait for transformation of a vector

    Trait for transformation of a vector

    Annotations
    @Since( "1.1.0" )
  13. class Word2Vec extends Serializable with Logging

    Word2Vec creates vector representation of words in a text corpus.

    Word2Vec creates vector representation of words in a text corpus. The algorithm first constructs a vocabulary from the corpus and then learns vector representation of words in the vocabulary. The vector representation can be used as features in natural language processing and machine learning algorithms.

    We used skip-gram model in our implementation and hierarchical softmax method to train the model. The variable names in the implementation matches the original C implementation.

    For original C implementation, see https://code.google.com/p/word2vec/ For research papers, see Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality.

    Annotations
    @Since( "1.1.0" )
  14. class Word2VecModel extends Serializable with Saveable

    Word2Vec model

    Word2Vec model

    Annotations
    @Since( "1.1.0" )

Value Members

  1. object ChiSqSelectorModel extends Loader[ChiSqSelectorModel] with Serializable
  2. object HashingTF extends Serializable
  3. object Word2VecModel extends Loader[Word2VecModel] with Serializable
    Annotations
    @Since( "1.4.0" )

Ungrouped