Package

org.apache.spark.mllib

feature

Permalink

package feature

Visibility
  1. Public
  2. All

Type Members

  1. class ChiSqSelector extends Serializable

    Permalink

    Creates a ChiSquared feature selector.

    Creates a ChiSquared feature selector. The selector supports different selection methods: numTopFeatures, percentile, fpr.

    • numTopFeatures chooses a fixed number of top features according to a chi-squared test.
    • percentile is similar but chooses a fraction of all features instead of a fixed number.
    • fpr chooses all features whose p-value is below a threshold, thus controlling the false positive rate of selection. By default, the selection method is numTopFeatures, with the default number of top features set to 50.
    Annotations
    @Since( "1.3.0" )
  2. class ChiSqSelectorModel extends VectorTransformer with Saveable

    Permalink

    Chi Squared selector model.

    Chi Squared selector model.

    Annotations
    @Since( "1.3.0" )
  3. class ElementwiseProduct extends VectorTransformer

    Permalink

    Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.

    Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.

    Annotations
    @Since( "1.4.0" )
  4. class HashingTF extends Serializable

    Permalink

    Maps a sequence of terms to their term frequencies using the hashing trick.

    Maps a sequence of terms to their term frequencies using the hashing trick.

    Annotations
    @Since( "1.1.0" )
  5. class IDF extends AnyRef

    Permalink

    Inverse document frequency (IDF).

    Inverse document frequency (IDF). The standard formulation is used: idf = log((m + 1) / (d(t) + 1)), where m is the total number of documents and d(t) is the number of documents that contain term t.

    This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable minDocFreq). For terms that are not in at least minDocFreq documents, the IDF is found as 0, resulting in TF-IDFs of 0.

    Annotations
    @Since( "1.1.0" )
  6. class IDFModel extends Serializable

    Permalink

    Represents an IDF model that can transform term frequency vectors.

    Represents an IDF model that can transform term frequency vectors.

    Annotations
    @Since( "1.1.0" )
  7. class Normalizer extends VectorTransformer

    Permalink

    Normalizes samples individually to unit Lp norm

    Normalizes samples individually to unit Lp norm

    For any 1 <= p < Double.PositiveInfinity, normalizes samples using sum(abs(vector).p)(1/p) as norm.

    For p = Double.PositiveInfinity, max(abs(vector)) will be used as norm for normalization.

    Annotations
    @Since( "1.1.0" )
  8. class PCA extends AnyRef

    Permalink

    A feature transformer that projects vectors to a low-dimensional space using PCA.

    A feature transformer that projects vectors to a low-dimensional space using PCA.

    Annotations
    @Since( "1.4.0" )
  9. class PCAModel extends VectorTransformer

    Permalink

    Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

    Model fitted by PCA that can project vectors to a low-dimensional space using PCA.

    Annotations
    @Since( "1.4.0" )
  10. class StandardScaler extends Logging

    Permalink

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.

    The "unit std" is computed using the corrected sample standard deviation (https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation), which is computed as the square root of the unbiased sample variance.

    Annotations
    @Since( "1.1.0" )
  11. class StandardScalerModel extends VectorTransformer

    Permalink

    Represents a StandardScaler model that can transform vectors.

    Represents a StandardScaler model that can transform vectors.

    Annotations
    @Since( "1.1.0" )
  12. trait VectorTransformer extends Serializable

    Permalink

    :: DeveloperApi :: Trait for transformation of a vector

    :: DeveloperApi :: Trait for transformation of a vector

    Annotations
    @Since( "1.1.0" ) @DeveloperApi()
  13. class Word2Vec extends Serializable with Logging

    Permalink

    Word2Vec creates vector representation of words in a text corpus.

    Word2Vec creates vector representation of words in a text corpus. The algorithm first constructs a vocabulary from the corpus and then learns vector representation of words in the vocabulary. The vector representation can be used as features in natural language processing and machine learning algorithms.

    We used skip-gram model in our implementation and hierarchical softmax method to train the model. The variable names in the implementation matches the original C implementation.

    For original C implementation, see https://code.google.com/p/word2vec/ For research papers, see Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality.

    Annotations
    @Since( "1.1.0" )
  14. class Word2VecModel extends Serializable with Saveable

    Permalink

    Word2Vec model

    Word2Vec model

    Annotations
    @Since( "1.1.0" )

Value Members

  1. object ChiSqSelectorModel extends Loader[ChiSqSelectorModel] with Serializable

    Permalink
  2. object HashingTF extends Serializable

    Permalink
  3. object Word2VecModel extends Loader[Word2VecModel] with Serializable

    Permalink
    Annotations
    @Since( "1.4.0" )

Ungrouped