public class LDA extends Estimator<LDAModel> implements LDAParams, DefaultParamsWritable
Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data
Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
Input data (featuresCol):
LDA is given a collection of documents as input data, via the featuresCol parameter.
Each document is specified as a Vector
of length vocabSize, where each entry is the
count for the corresponding term (word) in the document. Feature transformers such as
Tokenizer
and CountVectorizer
can be useful for converting text to word count vectors.
Modifier and Type | Method and Description |
---|---|
LDA |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
LDAModel |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
static LDA |
load(String path) |
static MLReader<LDA> |
read() |
LDA |
setCheckpointInterval(int value) |
LDA |
setDocConcentration(double value) |
LDA |
setDocConcentration(double[] value) |
LDA |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDA |
setK(int value) |
LDA |
setKeepLastCheckpoint(boolean value) |
LDA |
setLearningDecay(double value) |
LDA |
setLearningOffset(double value) |
LDA |
setMaxIter(int value) |
LDA |
setOptimizeDocConcentration(boolean value) |
LDA |
setOptimizer(String value) |
LDA |
setSeed(long value) |
LDA |
setSubsamplingRate(double value) |
LDA |
setTopicConcentration(double value) |
LDA |
setTopicDistributionCol(String value) |
StructType |
transformSchema(StructType schema)
:: DeveloperApi ::
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
docConcentration, getDocConcentration, getK, getKeepLastCheckpoint, getLearningDecay, getLearningOffset, getOldDocConcentration, getOldOptimizer, getOldTopicConcentration, getOptimizeDocConcentration, getOptimizer, getSubsamplingRate, getTopicConcentration, getTopicDistributionCol, k, keepLastCheckpoint, learningDecay, learningOffset, optimizeDocConcentration, optimizer, subsamplingRate, supportedOptimizers, topicConcentration, topicDistributionCol, validateAndTransformSchema
featuresCol, getFeaturesCol
getMaxIter, maxIter
checkpointInterval, getCheckpointInterval
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
toString
write
save
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
public static LDA load(String path)
public String uid()
Identifiable
uid
in interface Identifiable
public LDA setFeaturesCol(String value)
Vector
representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value
- (undocumented)public LDA setMaxIter(int value)
public LDA setSeed(long value)
public LDA setCheckpointInterval(int value)
public LDA setK(int value)
public LDA setDocConcentration(double[] value)
public LDA setDocConcentration(double value)
public LDA setTopicConcentration(double value)
public LDA setOptimizer(String value)
public LDA setTopicDistributionCol(String value)
public LDA setLearningOffset(double value)
public LDA setLearningDecay(double value)
public LDA setSubsamplingRate(double value)
public LDA setOptimizeDocConcentration(boolean value)
public LDA setKeepLastCheckpoint(boolean value)
public LDA copy(ParamMap extra)
Params
defaultCopy()
.public LDAModel fit(Dataset<?> dataset)
Estimator
public StructType transformSchema(StructType schema)
PipelineStage
Check transform validity and derive the output schema from the input schema.
We check validity for interactions between parameters during transformSchema
and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate()
.
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema
in class PipelineStage
schema
- (undocumented)