org.apache.spark.mllib.clustering

DistributedLDAModel

class DistributedLDAModel extends LDAModel

:: Experimental ::

Distributed LDA model. This model stores the inferred topics, the full training dataset, and the topic distributions. When computing topics for new documents, it may give more accurate answers than the LocalLDAModel.

Annotations
@Experimental()
Linear Supertypes
LDAModel, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. DistributedLDAModel
  2. LDAModel
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  8. def describeTopics(maxTermsPerTopic: Int): Array[(Array[Int], Array[Double])]

    Return the topics described by weighted terms.

    Return the topics described by weighted terms.

    This limits the number of terms per topic. This is approximate; it may not return exactly the top-weighted terms for each topic. To get a more precise set of top terms, increase maxTermsPerTopic.

    maxTermsPerTopic

    Maximum number of terms to collect for each topic.

    returns

    Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

    Definition Classes
    DistributedLDAModelLDAModel
  9. def describeTopics(): Array[(Array[Int], Array[Double])]

    Return the topics described by weighted terms.

    Return the topics described by weighted terms.

    WARNING: If vocabSize and k are large, this can return a large object!

    returns

    Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

    Definition Classes
    LDAModel
  10. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  11. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  12. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  13. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  14. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  16. def javaTopicDistributions: JavaPairRDD[Long, Vector]

    Java-friendly version of topicDistributions

  17. val k: Int

    Number of topics

    Number of topics

    Definition Classes
    DistributedLDAModelLDAModel
  18. lazy val logLikelihood: Double

    Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)

    Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)

    Note:

    • This excludes the prior; for that, use logPrior.
    • Even with logPrior, this is NOT the same as the data log likelihood given the hyperparameters.
  19. lazy val logPrior: Double

    Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)

  20. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  21. final def notify(): Unit

    Definition Classes
    AnyRef
  22. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  23. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  24. def toLocal: LocalLDAModel

    Convert model to a local model.

    Convert model to a local model. The local model stores the inferred topics but not the topic distributions for training documents.

  25. def toString(): String

    Definition Classes
    AnyRef → Any
  26. def topicDistributions: RDD[(Long, Vector)]

    For each document in the training set, return the distribution over topics for that document ("theta_doc").

    For each document in the training set, return the distribution over topics for that document ("theta_doc").

    returns

    RDD of (document ID, topic distribution) pairs

  27. lazy val topicsMatrix: Matrix

    Inferred topics, where each topic is represented by a distribution over terms.

    Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

    WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.

    Definition Classes
    DistributedLDAModelLDAModel
  28. val vocabSize: Int

    Vocabulary size (number of terms or terms in the vocabulary)

    Vocabulary size (number of terms or terms in the vocabulary)

    Definition Classes
    DistributedLDAModelLDAModel
  29. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from LDAModel

Inherited from AnyRef

Inherited from Any

Ungrouped