public abstract class LDAModel extends Model<LDAModel> implements LDAParams, org.apache.spark.internal.Logging, MLWritable
LDA.
param: vocabSize Vocabulary size (number of terms or words in the vocabulary) param: sparkSession Used to construct local DataFrames for returning query results
| Modifier and Type | Method and Description |
|---|---|
IntParam |
checkpointInterval()
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
|
Dataset<Row> |
describeTopics() |
Dataset<Row> |
describeTopics(int maxTermsPerTopic)
Return the topics described by their top-weighted terms.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Vector |
estimatedDocConcentration()
Value for
docConcentration estimated from data. |
Param<String> |
featuresCol()
Param for features column name.
|
abstract boolean |
isDistributed()
Indicates whether this instance is of type
DistributedLDAModel |
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
BooleanParam |
keepLastCheckpoint()
For EM optimizer only:
optimizer = "em". |
DoubleParam |
learningDecay()
For Online optimizer only:
optimizer = "online". |
DoubleParam |
learningOffset()
For Online optimizer only:
optimizer = "online". |
double |
logLikelihood(Dataset<?> dataset)
Calculates a lower bound on the log likelihood of the entire corpus.
|
double |
logPerplexity(Dataset<?> dataset)
Calculate an upper bound on perplexity.
|
IntParam |
maxIter()
Param for maximum number of iterations (>= 0).
|
BooleanParam |
optimizeDocConcentration()
For Online optimizer only (currently):
optimizer = "online". |
Param<String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
LongParam |
seed()
Param for random seed.
|
LDAModel |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDAModel |
setSeed(long value) |
LDAModel |
setTopicDistributionCol(String value) |
DoubleParam |
subsamplingRate()
For Online optimizer only:
optimizer = "online". |
String[] |
supportedOptimizers()
Supported values for Param
optimizer. |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
Matrix |
topicsMatrix()
Inferred topics, where each topic is represented by a distribution over terms.
|
Dataset<Row> |
transform(Dataset<?> dataset)
Transforms the input dataset.
|
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
int |
vocabSize() |
transform, transform, transformparamsequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetDocConcentration, getK, getKeepLastCheckpoint, getLearningDecay, getLearningOffset, getOldDocConcentration, getOldOptimizer, getOldTopicConcentration, getOptimizeDocConcentration, getOptimizer, getSubsamplingRate, getTopicConcentration, getTopicDistributionCol, validateAndTransformSchemagetFeaturesColgetMaxItergetCheckpointIntervalclear, copy, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoString$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitializesave, writepublic final IntParam checkpointInterval()
HasCheckpointIntervalcheckpointInterval in interface HasCheckpointIntervalpublic Dataset<Row> describeTopics(int maxTermsPerTopic)
maxTermsPerTopic - Maximum number of terms to collect for each topic.
Default value of 10.public final DoubleArrayParam docConcentration()
LDAParamsThis is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration vector must be length k.
(default = automatic)
Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.
docConcentration in interface LDAParamspublic Vector estimatedDocConcentration()
docConcentration estimated from data.
If Online LDA was used and optimizeDocConcentration was set to false,
then this returns the fixed (given) value for the docConcentration parameter.public final Param<String> featuresCol()
HasFeaturesColfeaturesCol in interface HasFeaturesColpublic abstract boolean isDistributed()
DistributedLDAModelpublic final IntParam k()
LDAParamspublic final BooleanParam keepLastCheckpoint()
LDAParamsoptimizer = "em".
If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.
See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and
DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.
Default: true
keepLastCheckpoint in interface LDAParamspublic final DoubleParam learningDecay()
LDAParamsoptimizer = "online".
Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.
learningDecay in interface LDAParamspublic final DoubleParam learningOffset()
LDAParamsoptimizer = "online".
A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.
learningOffset in interface LDAParamspublic double logLikelihood(Dataset<?> dataset)
See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
dataset - test corpus to use for calculating log likelihoodpublic double logPerplexity(Dataset<?> dataset)
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
dataset - test corpus to use for calculating perplexitypublic final IntParam maxIter()
HasMaxItermaxIter in interface HasMaxIterpublic final BooleanParam optimizeDocConcentration()
LDAParamsoptimizer = "online".
Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false
optimizeDocConcentration in interface LDAParamspublic final Param<String> optimizer()
LDAParamsFor details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here
public final LongParam seed()
HasSeedpublic LDAModel setFeaturesCol(String value)
Vector representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value - (undocumented)public LDAModel setSeed(long value)
public LDAModel setTopicDistributionCol(String value)
public final DoubleParam subsamplingRate()
LDAParamsoptimizer = "online".
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction greater than or equal to 1.
Note: This is the same as the miniBatchFraction parameter in
OnlineLDAOptimizer.
Default: 0.05, i.e., 5% of total documents.
subsamplingRate in interface LDAParamspublic final String[] supportedOptimizers()
LDAParamsoptimizer.supportedOptimizers in interface LDAParamspublic final DoubleParam topicConcentration()
LDAParamsThis is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.
topicConcentration in interface LDAParamspublic final Param<String> topicDistributionCol()
LDAParamsThis uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
topicDistributionCol in interface LDAParamspublic Matrix topicsMatrix()
WARNING: If this model is actually a DistributedLDAModel instance produced by
the Expectation-Maximization ("em") optimizer, then this method could involve
collecting a large amount of data to the driver (on the order of vocabSize x k).
public Dataset<Row> transform(Dataset<?> dataset)
WARNING: If this model is an instance of DistributedLDAModel (produced when optimizer
is set to "em"), this involves collecting a large topicsMatrix to the driver.
This implementation may be changed in the future.
transform in class Transformerdataset - (undocumented)public StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)public String uid()
Identifiableuid in interface Identifiablepublic int vocabSize()