package feature
 Alphabetic
 Public
 All
Type Members

class
ChiSqSelector extends Serializable
Creates a ChiSquared feature selector.
Creates a ChiSquared feature selector. The selector supports different selection methods:
numTopFeatures
,percentile
,fpr
,fdr
,fwe
.numTopFeatures
chooses a fixed number of top features according to a chisquared test.percentile
is similar but chooses a fraction of all features instead of a fixed number.fpr
chooses all features whose pvalues are below a threshold, thus controlling the false positive rate of selection.fdr
uses the [BenjaminiHochberg procedure] (https://en.wikipedia.org/wiki/False_discovery_rate#Benjamini.E2.80.93Hochberg_procedure) to choose all features whose false discovery rate is below a threshold.fwe
chooses all features whose pvalues are below a threshold. The threshold is scaled by 1/numFeatures, thus controlling the familywise error rate of selection. By default, the selection method isnumTopFeatures
, with the default number of top features set to 50.
 Annotations
 @Since( "1.3.0" )

class
ChiSqSelectorModel extends VectorTransformer with Saveable
Chi Squared selector model.
Chi Squared selector model.
 Annotations
 @Since( "1.3.0" )

class
ElementwiseProduct extends VectorTransformer
Outputs the Hadamard product (i.e., the elementwise product) of each input vector with a provided "weight" vector.
Outputs the Hadamard product (i.e., the elementwise product) of each input vector with a provided "weight" vector. In other words, it scales each column of the dataset by a scalar multiplier.
 Annotations
 @Since( "1.4.0" )

class
HashingTF extends Serializable
Maps a sequence of terms to their term frequencies using the hashing trick.
Maps a sequence of terms to their term frequencies using the hashing trick.
 Annotations
 @Since( "1.1.0" )

class
IDF extends AnyRef
Inverse document frequency (IDF).
Inverse document frequency (IDF). The standard formulation is used:
idf = log((m + 1) / (d(t) + 1))
, wherem
is the total number of documents andd(t)
is the number of documents that contain termt
.This implementation supports filtering out terms which do not appear in a minimum number of documents (controlled by the variable
minDocFreq
). For terms that are not in at leastminDocFreq
documents, the IDF is found as 0, resulting in TFIDFs of 0. The document frequency is 0 as well for such terms Annotations
 @Since( "1.1.0" )

class
IDFModel extends Serializable
Represents an IDF model that can transform term frequency vectors.
Represents an IDF model that can transform term frequency vectors.
 Annotations
 @Since( "1.1.0" )

class
Normalizer extends VectorTransformer
Normalizes samples individually to unit L^{p} norm
Normalizes samples individually to unit L^{p} norm
For any 1 <= p < Double.PositiveInfinity, normalizes samples using sum(abs(vector).^{p})^{(1/p)} as norm.
For p = Double.PositiveInfinity, max(abs(vector)) will be used as norm for normalization.
 Annotations
 @Since( "1.1.0" )

class
PCA extends AnyRef
A feature transformer that projects vectors to a lowdimensional space using PCA.
A feature transformer that projects vectors to a lowdimensional space using PCA.
 Annotations
 @Since( "1.4.0" )

class
PCAModel extends VectorTransformer
Model fitted by PCA that can project vectors to a lowdimensional space using PCA.
Model fitted by PCA that can project vectors to a lowdimensional space using PCA.
 Annotations
 @Since( "1.4.0" )

class
StandardScaler extends Logging
Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.
Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.
The "unit std" is computed using the corrected sample standard deviation (https://en.wikipedia.org/wiki/Standard_deviation#Corrected_sample_standard_deviation), which is computed as the square root of the unbiased sample variance.
 Annotations
 @Since( "1.1.0" )

class
StandardScalerModel extends VectorTransformer
Represents a StandardScaler model that can transform vectors.
Represents a StandardScaler model that can transform vectors.
 Annotations
 @Since( "1.1.0" )

trait
VectorTransformer extends Serializable
Trait for transformation of a vector
Trait for transformation of a vector
 Annotations
 @Since( "1.1.0" )

class
Word2Vec extends Serializable with Logging
Word2Vec creates vector representation of words in a text corpus.
Word2Vec creates vector representation of words in a text corpus. The algorithm first constructs a vocabulary from the corpus and then learns vector representation of words in the vocabulary. The vector representation can be used as features in natural language processing and machine learning algorithms.
We used skipgram model in our implementation and hierarchical softmax method to train the model. The variable names in the implementation matches the original C implementation.
For original C implementation, see https://code.google.com/p/word2vec/ For research papers, see Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality.
 Annotations
 @Since( "1.1.0" )

class
Word2VecModel extends Serializable with Saveable
Word2Vec model
Word2Vec model
 Annotations
 @Since( "1.1.0" )
Value Members
 object ChiSqSelectorModel extends Loader[ChiSqSelectorModel] with Serializable
 object HashingTF extends Serializable

object
Word2VecModel extends Loader[Word2VecModel] with Serializable
 Annotations
 @Since( "1.4.0" )