Package org.apache.spark.ml.clustering

Interface LDAParams

All Superinterfaces:
HasCheckpointInterval, HasFeaturesCol, HasMaxIter, HasSeed, Identifiable, Params, Serializable, scala.Serializable
All Known Implementing Classes:
DistributedLDAModel, LDA, LDAModel, LocalLDAModel
public interface LDAParams extends Params, HasFeaturesCol, HasMaxIter, HasSeed, HasCheckpointInterval

  • Method Details

    • k

      IntParam k()
      Param for the number of topics (clusters) to infer. Must be > 1. Default: 10.

      Returns:
      (undocumented)

    • getK

      int getK()

    • docConcentration

      DoubleArrayParam docConcentration()
      Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").

      This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).

      If not set by the user, then docConcentration is set automatically. If set to singleton vector [alpha], then alpha is replicated to a vector of length k in fitting. Otherwise, the docConcentration() vector must be length k. (default = automatic)

      Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.

      Returns:
      (undocumented)

    • getDocConcentration

      double[] getDocConcentration()

    • getOldDocConcentration

      Vector getOldDocConcentration()
      Get docConcentration used by spark.mllib LDA

    • topicConcentration

      DoubleParam topicConcentration()
      Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.

      This is the parameter to a symmetric Dirichlet distribution.

      Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.

      If not set by the user, then topicConcentration is set automatically. (default = automatic)

      Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.

      Returns:
      (undocumented)

    • getTopicConcentration

      double getTopicConcentration()

    • getOldTopicConcentration

      double getOldTopicConcentration()
      Get topicConcentration used by spark.mllib LDA

    • supportedOptimizers

      String[] supportedOptimizers()
      Supported values for Param optimizer().

    • optimizer

      Param<String> optimizer()
      Optimizer or inference algorithm used to estimate the LDA model. Currently supported (case-insensitive): - "online": Online Variational Bayes (default) - "em": Expectation-Maximization

      For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here

      Returns:
      (undocumented)

    • getOptimizer

      String getOptimizer()

    • topicDistributionCol

      Param<String> topicDistributionCol()
      Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.

      This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.

      Returns:
      (undocumented)

    • getTopicDistributionCol

      String getTopicDistributionCol()

    • learningOffset

      DoubleParam learningOffset()
      For Online optimizer only: optimizer() = "online".

      A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.

      Returns:
      (undocumented)

    • getLearningOffset

      double getLearningOffset()

    • learningDecay

      DoubleParam learningDecay()
      For Online optimizer only: optimizer() = "online".

      Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.

      Returns:
      (undocumented)

    • getLearningDecay

      double getLearningDecay()

    • subsamplingRate

      DoubleParam subsamplingRate()
      For Online optimizer only: optimizer() = "online".

      Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].

      Note that this should be adjusted in synch with LDA.maxIter so the entire corpus is used. Specifically, set both so that maxIterations * miniBatchFraction greater than or equal to 1.

      Note: This is the same as the miniBatchFraction parameter in OnlineLDAOptimizer.

      Default: 0.05, i.e., 5% of total documents.

      Returns:
      (undocumented)

    • getSubsamplingRate

      double getSubsamplingRate()

    • optimizeDocConcentration

      BooleanParam optimizeDocConcentration()
      For Online optimizer only (currently): optimizer() = "online".

      Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false

      Returns:
      (undocumented)

    • getOptimizeDocConcentration

      boolean getOptimizeDocConcentration()

    • keepLastCheckpoint

      BooleanParam keepLastCheckpoint()
      For EM optimizer only: optimizer() = "em".

      If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.

      See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.

      Default: true

      Returns:
      (undocumented)

    • getKeepLastCheckpoint

      boolean getKeepLastCheckpoint()

    • validateAndTransformSchema

      StructType validateAndTransformSchema(StructType schema)
      Validates and transforms the input schema.

      Parameters:
      schema - input schema
      Returns:
      output schema

    • getOldOptimizer

      LDAOptimizer getOldOptimizer()