Packages

  • package root
    Definition Classes
    root
  • package org
    Definition Classes
    root
  • package apache
    Definition Classes
    org
  • package spark

    Core Spark functionality.

    Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.

    In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.

    Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.

    Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.

    Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases.

    Definition Classes
    apache
  • package mllib

    RDD-based machine learning APIs (in maintenance mode).

    RDD-based machine learning APIs (in maintenance mode).

    The spark.mllib package is in maintenance mode as of the Spark 2.0.0 release to encourage migration to the DataFrame-based APIs under the org.apache.spark.ml package. While in maintenance mode,

    • no new features in the RDD-based spark.mllib package will be accepted, unless they block implementing new features in the DataFrame-based spark.ml package;
    • bug fixes in the RDD-based APIs will still be accepted.

    The developers will continue adding more features to the DataFrame-based APIs in the 2.x series to reach feature parity with the RDD-based APIs. And once we reach feature parity, this package will be deprecated.

    Definition Classes
    spark
    See also

    SPARK-4591 to track the progress of feature parity

  • package linalg
    Definition Classes
    mllib
  • package distributed
    Definition Classes
    linalg
  • DenseMatrix
  • DenseVector
  • Matrices
  • Matrix
  • QRDecomposition
  • SingularValueDecomposition
  • SparseMatrix
  • SparseVector
  • Vector
  • VectorUDT
  • Vectors

class SparseMatrix extends Matrix

Column-major sparse matrix. The entry values are stored in Compressed Sparse Column (CSC) format. For example, the following matrix

1.0 0.0 4.0
0.0 3.0 5.0
2.0 0.0 6.0

is stored as values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0], rowIndices=[0, 2, 1, 0, 1, 2], colPointers=[0, 2, 3, 6].

Annotations
@Since("1.2.0") @SQLUserDefinedType()
Source
Matrices.scala
Linear Supertypes
Matrix, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparseMatrix
  2. Matrix
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SparseMatrix(numRows: Int, numCols: Int, colPtrs: Array[Int], rowIndices: Array[Int], values: Array[Double])

    Column-major sparse matrix.

    Column-major sparse matrix. The entry values are stored in Compressed Sparse Column (CSC) format. For example, the following matrix

    1.0 0.0 4.0
0.0 3.0 5.0
2.0 0.0 6.0

    is stored as values: [1.0, 2.0, 3.0, 4.0, 5.0, 6.0], rowIndices=[0, 2, 1, 0, 1, 2], colPointers=[0, 2, 3, 6].

    numRows

    number of rows

    numCols

    number of columns

    colPtrs

    the index corresponding to the start of a new column

    rowIndices

    the row index of the entry. They must be in strictly increasing order for each column

    values

    non-zero matrix entries in column major

    Annotations
    @Since("1.2.0")
  2. new SparseMatrix(numRows: Int, numCols: Int, colPtrs: Array[Int], rowIndices: Array[Int], values: Array[Double], isTransposed: Boolean)

    numRows

    number of rows

    numCols

    number of columns

    colPtrs

    the index corresponding to the start of a new column (if not transposed)

    rowIndices

    the row index of the entry (if not transposed). They must be in strictly increasing order for each column

    values

    nonzero matrix entries in column major (if not transposed)

    isTransposed

    whether the matrix is transposed. If true, the matrix can be considered Compressed Sparse Row (CSR) format, where colPtrs behaves as rowPtrs, and rowIndices behave as colIndices, and values are stored in row major.

    Annotations
    @Since("1.3.0")

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def apply(i: Int, j: Int): Double

    Gets the (i, j)-th element.

    Gets the (i, j)-th element.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.3.0")
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def asML: ml.linalg.SparseMatrix

    Convert this matrix to the new mllib-local representation.

    Convert this matrix to the new mllib-local representation. This does NOT copy the data; it copies references.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("2.0.0")
  7. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  8. def colIter: Iterator[Vector]

    Returns an iterator of column vectors.

    Returns an iterator of column vectors. This operation could be expensive, depending on the underlying storage.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("2.0.0")
  9. val colPtrs: Array[Int]
    Annotations
    @Since("1.2.0")
  10. def copy: SparseMatrix

    Get a deep copy of the matrix.

    Get a deep copy of the matrix.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.4.0")
  11. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  12. def equals(o: Any): Boolean
    Definition Classes
    SparseMatrix → AnyRef → Any
  13. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  14. def hashCode(): Int
    Definition Classes
    SparseMatrix → AnyRef → Any
  15. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  16. val isTransposed: Boolean

    Flag that keeps track whether the matrix is transposed or not.

    Flag that keeps track whether the matrix is transposed or not. False by default.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.3.0")
  17. def multiply(y: Vector): DenseVector

    Convenience method for Matrix-Vector multiplication.

    Convenience method for Matrix-Vector multiplication.

    Definition Classes
    Matrix
    Annotations
    @Since("1.4.0")
  18. def multiply(y: DenseVector): DenseVector

    Convenience method for Matrix-DenseVector multiplication.

    Convenience method for Matrix-DenseVector multiplication. For binary compatibility.

    Definition Classes
    Matrix
    Annotations
    @Since("1.2.0")
  19. def multiply(y: DenseMatrix): DenseMatrix

    Convenience method for Matrix-DenseMatrix multiplication.

    Convenience method for Matrix-DenseMatrix multiplication.

    Definition Classes
    Matrix
    Annotations
    @Since("1.2.0")
  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  23. def numActives: Int

    Find the number of values stored explicitly.

    Find the number of values stored explicitly. These values can be zero as well.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.5.0")
  24. val numCols: Int

    Number of columns.

    Number of columns.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.2.0")
  25. def numNonzeros: Int

    Find the number of non-zero active values.

    Find the number of non-zero active values.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.5.0")
  26. val numRows: Int

    Number of rows.

    Number of rows.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.2.0")
  27. val rowIndices: Array[Int]
    Annotations
    @Since("1.2.0")
  28. def rowIter: Iterator[Vector]

    Returns an iterator of row vectors.

    Returns an iterator of row vectors. This operation could be expensive, depending on the underlying storage.

    Definition Classes
    Matrix
    Annotations
    @Since("2.0.0")
  29. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  30. def toArray: Array[Double]

    Converts to a dense array in column major.

    Converts to a dense array in column major.

    Definition Classes
    Matrix
    Annotations
    @Since("1.0.0")
  31. def toDense: DenseMatrix

    Generate a DenseMatrix from the given SparseMatrix.

    Generate a DenseMatrix from the given SparseMatrix. The new matrix will have isTransposed set to false.

    Annotations
    @Since("1.3.0")
  32. def toString(maxLines: Int, maxLineWidth: Int): String

    A human readable representation of the matrix with maximum lines and width

    A human readable representation of the matrix with maximum lines and width

    Definition Classes
    Matrix
    Annotations
    @Since("1.4.0")
  33. def toString(): String

    A human readable representation of the matrix

    A human readable representation of the matrix

    Definition Classes
    Matrix → AnyRef → Any
  34. def transpose: SparseMatrix

    Transpose the Matrix.

    Transpose the Matrix. Returns a new Matrix instance sharing the same underlying data.

    Definition Classes
    SparseMatrix → Matrix
    Annotations
    @Since("1.3.0")
  35. val values: Array[Double]
    Annotations
    @Since("1.2.0")
  36. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  37. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  38. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Matrix

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped