Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark

    Core Spark functionality.

    Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.

    In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.

    Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.

    Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.

    Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases.

    Definition Classes
    apache
  • package mllib

    RDD-based machine learning APIs (in maintenance mode).

    RDD-based machine learning APIs (in maintenance mode).

    The spark.mllib package is in maintenance mode as of the Spark 2.0.0 release to encourage migration to the DataFrame-based APIs under the org.apache.spark.ml package. While in maintenance mode,

    • no new features in the RDD-based spark.mllib package will be accepted, unless they block implementing new features in the DataFrame-based spark.ml package;
    • bug fixes in the RDD-based APIs will still be accepted.

    The developers will continue adding more features to the DataFrame-based APIs in the 2.x series to reach feature parity with the RDD-based APIs. And once we reach feature parity, this package will be deprecated.

    Definition Classes
    spark
    See also

    SPARK-4591 to track the progress of feature parity

  • package util
    Definition Classes
    mllib
  • DataValidators
  • KMeansDataGenerator
  • LinearDataGenerator
  • Loader
  • LogisticRegressionDataGenerator
  • MFDataGenerator
  • MLUtils
  • SVMDataGenerator
  • Saveable

object MLUtils extends Logging

Helper methods to load, save and pre-process data used in MLLib.

Annotations
@Since( "0.8.0" )
Source
MLUtils.scala
Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MLUtils
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def appendBias(vector: Vector): Vector

    Returns a new vector with 1.0 (bias) appended to the input vector.

    Returns a new vector with 1.0 (bias) appended to the input vector.

    Annotations
    @Since( "1.0.0" )
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @IntrinsicCandidate()
  7. def convertMatrixColumnsFromML(dataset: Dataset[_], cols: String*): DataFrame

    Converts matrix columns in an input Dataset to the org.apache.spark.mllib.linalg.Matrix type from the new org.apache.spark.ml.linalg.Matrix type under the spark.ml package.

    Converts matrix columns in an input Dataset to the org.apache.spark.mllib.linalg.Matrix type from the new org.apache.spark.ml.linalg.Matrix type under the spark.ml package.

    dataset

    input dataset

    cols

    a list of matrix columns to be converted. Old matrix columns will be ignored. If unspecified, all new matrix columns will be converted except nested ones.

    returns

    the input DataFrame with new matrix columns converted to the old matrix type

    Annotations
    @Since( "2.0.0" ) @varargs()
  8. def convertMatrixColumnsToML(dataset: Dataset[_], cols: String*): DataFrame

    Converts Matrix columns in an input Dataset from the org.apache.spark.mllib.linalg.Matrix type to the new org.apache.spark.ml.linalg.Matrix type under the spark.ml package.

    Converts Matrix columns in an input Dataset from the org.apache.spark.mllib.linalg.Matrix type to the new org.apache.spark.ml.linalg.Matrix type under the spark.ml package.

    dataset

    input dataset

    cols

    a list of matrix columns to be converted. New matrix columns will be ignored. If unspecified, all old matrix columns will be converted except nested ones.

    returns

    the input DataFrame with old matrix columns converted to the new matrix type

    Annotations
    @Since( "2.0.0" ) @varargs()
  9. def convertVectorColumnsFromML(dataset: Dataset[_], cols: String*): DataFrame

    Converts vector columns in an input Dataset to the org.apache.spark.mllib.linalg.Vector type from the new org.apache.spark.ml.linalg.Vector type under the spark.ml package.

    Converts vector columns in an input Dataset to the org.apache.spark.mllib.linalg.Vector type from the new org.apache.spark.ml.linalg.Vector type under the spark.ml package.

    dataset

    input dataset

    cols

    a list of vector columns to be converted. Old vector columns will be ignored. If unspecified, all new vector columns will be converted except nested ones.

    returns

    the input DataFrame with new vector columns converted to the old vector type

    Annotations
    @Since( "2.0.0" ) @varargs()
  10. def convertVectorColumnsToML(dataset: Dataset[_], cols: String*): DataFrame

    Converts vector columns in an input Dataset from the org.apache.spark.mllib.linalg.Vector type to the new org.apache.spark.ml.linalg.Vector type under the spark.ml package.

    Converts vector columns in an input Dataset from the org.apache.spark.mllib.linalg.Vector type to the new org.apache.spark.ml.linalg.Vector type under the spark.ml package.

    dataset

    input dataset

    cols

    a list of vector columns to be converted. New vector columns will be ignored. If unspecified, all old vector columns will be converted except nested ones.

    returns

    the input DataFrame with old vector columns converted to the new vector type

    Annotations
    @Since( "2.0.0" ) @varargs()
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  13. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  14. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @IntrinsicCandidate()
  15. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  16. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  19. def kFold(df: DataFrame, numFolds: Int, foldColName: String): Array[(RDD[Row], RDD[Row])]

    Version of kFold() taking a fold column name.

    Version of kFold() taking a fold column name.

    Annotations
    @Since( "3.1.0" )
  20. def kFold[T](rdd: RDD[T], numFolds: Int, seed: Long)(implicit arg0: ClassTag[T]): Array[(RDD[T], RDD[T])]

    Version of kFold() taking a Long seed.

    Version of kFold() taking a Long seed.

    Annotations
    @Since( "2.0.0" )
  21. def kFold[T](rdd: RDD[T], numFolds: Int, seed: Int)(implicit arg0: ClassTag[T]): Array[(RDD[T], RDD[T])]

    Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data.

    Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data. Where k=numFolds.

    Annotations
    @Since( "1.0.0" )
  22. def loadLabeledPoints(sc: SparkContext, dir: String): RDD[LabeledPoint]

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of partitions.

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of partitions.

    Annotations
    @Since( "1.1.0" )
  23. def loadLabeledPoints(sc: SparkContext, path: String, minPartitions: Int): RDD[LabeledPoint]

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    minPartitions

    min number of partitions

    returns

    labeled points stored as an RDD[LabeledPoint]

    Annotations
    @Since( "1.1.0" )
  24. def loadLibSVMFile(sc: SparkContext, path: String): RDD[LabeledPoint]

    Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.

    Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.

    Annotations
    @Since( "1.0.0" )
  25. def loadLibSVMFile(sc: SparkContext, path: String, numFeatures: Int): RDD[LabeledPoint]

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.

    Annotations
    @Since( "1.0.0" )
  26. def loadLibSVMFile(sc: SparkContext, path: String, numFeatures: Int, minPartitions: Int): RDD[LabeledPoint]

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint].

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint]. The LIBSVM format is a text-based format used by LIBSVM and LIBLINEAR. Each line represents a labeled sparse feature vector using the following format:

    label index1:value1 index2:value2 ...

    where the indices are one-based and in ascending order. This method parses each line into a org.apache.spark.mllib.regression.LabeledPoint, where the feature indices are converted to zero-based.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    numFeatures

    number of features, which will be determined from the input data if a nonpositive value is given. This is useful when the dataset is already split into multiple files and you want to load them separately, because some features may not present in certain files, which leads to inconsistent feature dimensions.

    minPartitions

    min number of partitions

    returns

    labeled data stored as an RDD[LabeledPoint]

    Annotations
    @Since( "1.0.0" )
  27. def loadVectors(sc: SparkContext, path: String): RDD[Vector]

    Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.

    Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.

    Annotations
    @Since( "1.1.0" )
  28. def loadVectors(sc: SparkContext, path: String, minPartitions: Int): RDD[Vector]

    Loads vectors saved using RDD[Vector].saveAsTextFile.

    Loads vectors saved using RDD[Vector].saveAsTextFile.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    minPartitions

    min number of partitions

    returns

    vectors stored as an RDD[Vector]

    Annotations
    @Since( "1.1.0" )
  29. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  30. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  37. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  39. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  40. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  41. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  42. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  43. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @IntrinsicCandidate()
  44. def saveAsLibSVMFile(data: RDD[LabeledPoint], dir: String): Unit

    Save labeled data in LIBSVM format.

    Save labeled data in LIBSVM format.

    data

    an RDD of LabeledPoint to be saved

    dir

    directory to save the data

    Annotations
    @Since( "1.0.0" )
    See also

    org.apache.spark.mllib.util.MLUtils.loadLibSVMFile

  45. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  46. def toString(): String
    Definition Classes
    AnyRef → Any
  47. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  48. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  49. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped