public class LDA extends Estimator<LDAModel> implements LDAParams, DefaultParamsWritable
Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data
Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
Input data (featuresCol):
LDA is given a collection of documents as input data, via the featuresCol parameter.
Each document is specified as a Vector
of length vocabSize, where each entry is the
count for the corresponding term (word) in the document. Feature transformers such as
Tokenizer
and CountVectorizer
can be useful for converting text to word count vectors.
Modifier and Type | Method and Description |
---|---|
IntParam |
checkpointInterval()
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
|
LDA |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Param<String> |
featuresCol()
Param for features column name.
|
LDAModel |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
BooleanParam |
keepLastCheckpoint()
For EM optimizer only:
optimizer = "em". |
DoubleParam |
learningDecay()
For Online optimizer only:
optimizer = "online". |
DoubleParam |
learningOffset()
For Online optimizer only:
optimizer = "online". |
static LDA |
load(String path) |
IntParam |
maxIter()
Param for maximum number of iterations (>= 0).
|
BooleanParam |
optimizeDocConcentration()
For Online optimizer only (currently):
optimizer = "online". |
Param<String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
static MLReader<LDA> |
read() |
LongParam |
seed()
Param for random seed.
|
LDA |
setCheckpointInterval(int value) |
LDA |
setDocConcentration(double value) |
LDA |
setDocConcentration(double[] value) |
LDA |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDA |
setK(int value) |
LDA |
setKeepLastCheckpoint(boolean value) |
LDA |
setLearningDecay(double value) |
LDA |
setLearningOffset(double value) |
LDA |
setMaxIter(int value) |
LDA |
setOptimizeDocConcentration(boolean value) |
LDA |
setOptimizer(String value) |
LDA |
setSeed(long value) |
LDA |
setSubsamplingRate(double value) |
LDA |
setTopicConcentration(double value) |
LDA |
setTopicDistributionCol(String value) |
DoubleParam |
subsamplingRate()
For Online optimizer only:
optimizer = "online". |
String[] |
supportedOptimizers()
Supported values for Param
optimizer . |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
params
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
getDocConcentration, getK, getKeepLastCheckpoint, getLearningDecay, getLearningOffset, getOldDocConcentration, getOldOptimizer, getOldTopicConcentration, getOptimizeDocConcentration, getOptimizer, getSubsamplingRate, getTopicConcentration, getTopicDistributionCol, validateAndTransformSchema
getFeaturesCol
getMaxIter
getCheckpointInterval
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
toString
write
save
$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitialize
public static LDA load(String path)
public final IntParam k()
LDAParams
public final DoubleArrayParam docConcentration()
LDAParams
This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration
vector must be length k.
(default = automatic)
Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.
docConcentration
in interface LDAParams
public final DoubleParam topicConcentration()
LDAParams
This is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.
topicConcentration
in interface LDAParams
public final String[] supportedOptimizers()
LDAParams
optimizer
.supportedOptimizers
in interface LDAParams
public final Param<String> optimizer()
LDAParams
For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here
public final Param<String> topicDistributionCol()
LDAParams
This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
topicDistributionCol
in interface LDAParams
public final DoubleParam learningOffset()
LDAParams
optimizer
= "online".
A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.
learningOffset
in interface LDAParams
public final DoubleParam learningDecay()
LDAParams
optimizer
= "online".
Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.
learningDecay
in interface LDAParams
public final DoubleParam subsamplingRate()
LDAParams
optimizer
= "online".
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction greater than or equal to 1.
Note: This is the same as the miniBatchFraction
parameter in
OnlineLDAOptimizer
.
Default: 0.05, i.e., 5% of total documents.
subsamplingRate
in interface LDAParams
public final BooleanParam optimizeDocConcentration()
LDAParams
optimizer
= "online".
Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false
optimizeDocConcentration
in interface LDAParams
public final BooleanParam keepLastCheckpoint()
LDAParams
optimizer
= "em".
If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.
See DistributedLDAModel.getCheckpointFiles
for getting remaining checkpoints and
DistributedLDAModel.deleteCheckpointFiles
for removing remaining checkpoints.
Default: true
keepLastCheckpoint
in interface LDAParams
public final IntParam checkpointInterval()
HasCheckpointInterval
checkpointInterval
in interface HasCheckpointInterval
public final LongParam seed()
HasSeed
public final IntParam maxIter()
HasMaxIter
maxIter
in interface HasMaxIter
public final Param<String> featuresCol()
HasFeaturesCol
featuresCol
in interface HasFeaturesCol
public String uid()
Identifiable
uid
in interface Identifiable
public LDA setFeaturesCol(String value)
Vector
representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value
- (undocumented)public LDA setMaxIter(int value)
public LDA setSeed(long value)
public LDA setCheckpointInterval(int value)
public LDA setK(int value)
public LDA setDocConcentration(double[] value)
public LDA setDocConcentration(double value)
public LDA setTopicConcentration(double value)
public LDA setOptimizer(String value)
public LDA setTopicDistributionCol(String value)
public LDA setLearningOffset(double value)
public LDA setLearningDecay(double value)
public LDA setSubsamplingRate(double value)
public LDA setOptimizeDocConcentration(boolean value)
public LDA setKeepLastCheckpoint(boolean value)
public LDA copy(ParamMap extra)
Params
defaultCopy()
.public LDAModel fit(Dataset<?> dataset)
Estimator
public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema
and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate()
.
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema
in class PipelineStage
schema
- (undocumented)