Package org.apache.spark.ml.clustering

Class LDA

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Estimator<LDAModel>
org.apache.spark.ml.clustering.LDA
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, org.apache.spark.ml.clustering.LDAParams, Params, HasCheckpointInterval, HasFeaturesCol, HasMaxIter, HasSeed, DefaultParamsWritable, Identifiable, MLWritable
public class LDA extends Estimator<LDAModel> implements org.apache.spark.ml.clustering.LDAParams, DefaultParamsWritable
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.

Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data

Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.

Input data (featuresCol): LDA is given a collection of documents as input data, via the featuresCol parameter. Each document is specified as a Vector of length vocabSize, where each entry is the count for the corresponding term (word) in the document. Feature transformers such as Tokenizer and CountVectorizer can be useful for converting text to word count vectors.

See Also:

  • Constructor Details

    • LDA

      public LDA(String uid)

    • LDA

      public LDA()

  • Method Details

    • read

      public static MLReader<LDA> read()

    • load

      public static LDA load(String path)

    • k

      public final IntParam k()
      Specified by:
      k in interface org.apache.spark.ml.clustering.LDAParams

    • docConcentration

      public final DoubleArrayParam docConcentration()
      Specified by:
      docConcentration in interface org.apache.spark.ml.clustering.LDAParams

    • topicConcentration

      public final DoubleParam topicConcentration()
      Specified by:
      topicConcentration in interface org.apache.spark.ml.clustering.LDAParams

    • supportedOptimizers

      public final String[] supportedOptimizers()
      Specified by:
      supportedOptimizers in interface org.apache.spark.ml.clustering.LDAParams

    • optimizer

      public final Param<String> optimizer()
      Specified by:
      optimizer in interface org.apache.spark.ml.clustering.LDAParams

    • topicDistributionCol

      public final Param<String> topicDistributionCol()
      Specified by:
      topicDistributionCol in interface org.apache.spark.ml.clustering.LDAParams

    • learningOffset

      public final DoubleParam learningOffset()
      Specified by:
      learningOffset in interface org.apache.spark.ml.clustering.LDAParams

    • learningDecay

      public final DoubleParam learningDecay()
      Specified by:
      learningDecay in interface org.apache.spark.ml.clustering.LDAParams

    • subsamplingRate

      public final DoubleParam subsamplingRate()
      Specified by:
      subsamplingRate in interface org.apache.spark.ml.clustering.LDAParams

    • optimizeDocConcentration

      public final BooleanParam optimizeDocConcentration()
      Specified by:
      optimizeDocConcentration in interface org.apache.spark.ml.clustering.LDAParams

    • keepLastCheckpoint

      public final BooleanParam keepLastCheckpoint()
      Specified by:
      keepLastCheckpoint in interface org.apache.spark.ml.clustering.LDAParams

    • checkpointInterval

      public final IntParam checkpointInterval()
      Description copied from interface: HasCheckpointInterval
      Param for set checkpoint interval (&gt;= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.
      Specified by:
      checkpointInterval in interface HasCheckpointInterval
      Returns:
      (undocumented)

    • seed

      public final LongParam seed()
      Description copied from interface: HasSeed
      Param for random seed.
      Specified by:
      seed in interface HasSeed
      Returns:
      (undocumented)

    • maxIter

      public final IntParam maxIter()
      Description copied from interface: HasMaxIter
      Param for maximum number of iterations (&gt;= 0).
      Specified by:
      maxIter in interface HasMaxIter
      Returns:
      (undocumented)

    • featuresCol

      public final Param<String> featuresCol()
      Description copied from interface: HasFeaturesCol
      Param for features column name.
      Specified by:
      featuresCol in interface HasFeaturesCol
      Returns:
      (undocumented)

    • uid

      public String uid()
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Specified by:
      uid in interface Identifiable
      Returns:
      (undocumented)

    • setFeaturesCol

      public LDA setFeaturesCol(String value)
      The features for LDA should be a Vector representing the word counts in a document. The vector should be of length vocabSize, with counts for each term (word).

      Parameters:
      value - (undocumented)
      Returns:
      (undocumented)

    • setMaxIter

      public LDA setMaxIter(int value)

    • setSeed

      public LDA setSeed(long value)

    • setCheckpointInterval

      public LDA setCheckpointInterval(int value)

    • setK

      public LDA setK(int value)

    • setDocConcentration

      public LDA setDocConcentration(double[] value)

    • setDocConcentration

      public LDA setDocConcentration(double value)

    • setTopicConcentration

      public LDA setTopicConcentration(double value)

    • setOptimizer

      public LDA setOptimizer(String value)

    • setTopicDistributionCol

      public LDA setTopicDistributionCol(String value)

    • setLearningOffset

      public LDA setLearningOffset(double value)

    • setLearningDecay

      public LDA setLearningDecay(double value)

    • setSubsamplingRate

      public LDA setSubsamplingRate(double value)

    • setOptimizeDocConcentration

      public LDA setOptimizeDocConcentration(boolean value)

    • setKeepLastCheckpoint

      public LDA setKeepLastCheckpoint(boolean value)

    • copy

      public LDA copy(ParamMap extra)
      Description copied from interface: Params
      Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
      Specified by:
      copy in interface Params
      Specified by:
      copy in class Estimator<LDAModel>
      Parameters:
      extra - (undocumented)
      Returns:
      (undocumented)

    • fit

      public LDAModel fit(Dataset<?> dataset)
      Description copied from class: Estimator
      Fits a model to the input data.
      Specified by:
      fit in class Estimator<LDAModel>
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Specified by:
      transformSchema in class PipelineStage
      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)