org.apache.spark.mllib.util

MLUtils

object MLUtils

Helper methods to load, save and pre-process data used in ML Lib.

Annotations
@Since( "0.8.0" )
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. MLUtils
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def appendBias(vector: Vector): Vector

    Returns a new vector with 1.0 (bias) appended to the input vector.

    Returns a new vector with 1.0 (bias) appended to the input vector.

    Annotations
    @Since( "1.0.0" )
  7. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  10. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  12. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  13. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  14. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  15. def kFold[T](rdd: RDD[T], numFolds: Int, seed: Int)(implicit arg0: ClassTag[T]): Array[(RDD[T], RDD[T])]

    :: Experimental :: Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data.

    :: Experimental :: Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data. Where k=numFolds.

    Annotations
    @Since( "1.0.0" ) @Experimental()
  16. def loadLabeledPoints(sc: SparkContext, dir: String): RDD[LabeledPoint]

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of partitions.

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of partitions.

    Annotations
    @Since( "1.1.0" )
  17. def loadLabeledPoints(sc: SparkContext, path: String, minPartitions: Int): RDD[LabeledPoint]

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.

    Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    minPartitions

    min number of partitions

    returns

    labeled points stored as an RDD[LabeledPoint]

    Annotations
    @Since( "1.1.0" )
  18. def loadLibSVMFile(sc: SparkContext, path: String): RDD[LabeledPoint]

    Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.

    Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.

    Annotations
    @Since( "1.0.0" )
  19. def loadLibSVMFile(sc: SparkContext, path: String, numFeatures: Int): RDD[LabeledPoint]

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.

    Annotations
    @Since( "1.0.0" )
  20. def loadLibSVMFile(sc: SparkContext, path: String, numFeatures: Int, minPartitions: Int): RDD[LabeledPoint]

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint].

    Loads labeled data in the LIBSVM format into an RDD[LabeledPoint]. The LIBSVM format is a text-based format used by LIBSVM and LIBLINEAR. Each line represents a labeled sparse feature vector using the following format:

    label index1:value1 index2:value2 ...

    where the indices are one-based and in ascending order. This method parses each line into a org.apache.spark.mllib.regression.LabeledPoint, where the feature indices are converted to zero-based.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    numFeatures

    number of features, which will be determined from the input data if a nonpositive value is given. This is useful when the dataset is already split into multiple files and you want to load them separately, because some features may not present in certain files, which leads to inconsistent feature dimensions.

    minPartitions

    min number of partitions

    returns

    labeled data stored as an RDD[LabeledPoint]

    Annotations
    @Since( "1.0.0" )
  21. def loadVectors(sc: SparkContext, path: String): RDD[Vector]

    Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.

    Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.

    Annotations
    @Since( "1.1.0" )
  22. def loadVectors(sc: SparkContext, path: String, minPartitions: Int): RDD[Vector]

    Loads vectors saved using RDD[Vector].saveAsTextFile.

    Loads vectors saved using RDD[Vector].saveAsTextFile.

    sc

    Spark context

    path

    file or directory path in any Hadoop-supported file system URI

    minPartitions

    min number of partitions

    returns

    vectors stored as an RDD[Vector]

    Annotations
    @Since( "1.1.0" )
  23. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. final def notify(): Unit

    Definition Classes
    AnyRef
  25. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  26. def saveAsLibSVMFile(data: RDD[LabeledPoint], dir: String): Unit

    Save labeled data in LIBSVM format.

    Save labeled data in LIBSVM format.

    data

    an RDD of LabeledPoint to be saved

    dir

    directory to save the data

    Annotations
    @Since( "1.0.0" )
    See also

    org.apache.spark.mllib.util.MLUtils#loadLibSVMFile

  27. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  28. def toString(): String

    Definition Classes
    AnyRef → Any
  29. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def loadLabeledData(sc: SparkContext, dir: String): RDD[LabeledPoint]

    Load labeled data from a file.

    Load labeled data from a file. The data format used here is L, f1 f2 ... where f1, f2 are feature values in Double and L is the corresponding label as Double.

    sc

    SparkContext

    dir

    Directory to the input data files.

    returns

    An RDD of LabeledPoint. Each labeled point has two elements: the first element is the label, and the second element represents the feature values (an array of Double).

    Annotations
    @Since( "1.0.0" ) @deprecated
    Deprecated

    (Since version 1.0.1) Should use MLUtils.loadLabeledPoints instead.

  2. def loadLibSVMFile(sc: SparkContext, path: String, multiclass: Boolean): RDD[LabeledPoint]

    Annotations
    @Since( "1.0.0" ) @deprecated
    Deprecated

    (Since version 1.1.0) use method without multiclass argument, which no longer has effect

  3. def loadLibSVMFile(sc: SparkContext, path: String, multiclass: Boolean, numFeatures: Int): RDD[LabeledPoint]

    Annotations
    @Since( "1.0.0" ) @deprecated
    Deprecated

    (Since version 1.1.0) use method without multiclass argument, which no longer has effect

  4. def loadLibSVMFile(sc: SparkContext, path: String, multiclass: Boolean, numFeatures: Int, minPartitions: Int): RDD[LabeledPoint]

    Annotations
    @Since( "1.0.0" ) @deprecated
    Deprecated

    (Since version 1.1.0) use method without multiclass argument, which no longer has effect

  5. def saveLabeledData(data: RDD[LabeledPoint], dir: String): Unit

    Save labeled data to a file.

    Save labeled data to a file. The data format used here is L, f1 f2 ... where f1, f2 are feature values in Double and L is the corresponding label as Double.

    data

    An RDD of LabeledPoints containing data to be saved.

    dir

    Directory to save the data.

    Annotations
    @Since( "1.0.0" ) @deprecated
    Deprecated

    (Since version 1.0.1) Should use RDD[LabeledPoint].saveAsTextFile instead.

Inherited from AnyRef

Inherited from Any

Ungrouped