Package

org.apache.spark.mllib

regression

Permalink

package regression

Visibility
  1. Public
  2. All

Type Members

  1. abstract class GeneralizedLinearAlgorithm[M <: GeneralizedLinearModel] extends Logging with Serializable

    Permalink

    :: DeveloperApi :: GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM).

    :: DeveloperApi :: GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM). This class should be extended with an Optimizer to create a new GLM.

    Annotations
    @Since( "0.8.0" ) @DeveloperApi()
  2. abstract class GeneralizedLinearModel extends Serializable

    Permalink

    :: DeveloperApi :: GeneralizedLinearModel (GLM) represents a model trained using GeneralizedLinearAlgorithm.

    :: DeveloperApi :: GeneralizedLinearModel (GLM) represents a model trained using GeneralizedLinearAlgorithm. GLMs consist of a weight vector and an intercept.

    Annotations
    @Since( "0.8.0" ) @DeveloperApi()
  3. class IsotonicRegression extends Serializable

    Permalink

    Isotonic regression.

    Isotonic regression. Currently implemented using parallelized pool adjacent violators algorithm. Only univariate (single feature) algorithm supported.

    Sequential PAV implementation based on: Tibshirani, Ryan J., Holger Hoefling, and Robert Tibshirani. "Nearly-isotonic regression." Technometrics 53.1 (2011): 54-61. Available from http://www.stat.cmu.edu/~ryantibs/papers/neariso.pdf

    Sequential PAV parallelization based on: Kearsley, Anthony J., Richard A. Tapia, and Michael W. Trosset. "An approach to parallelizing isotonic regression." Applied Mathematics and Parallel Computing. Physica-Verlag HD, 1996. 141-147. Available from http://softlib.rice.edu/pub/CRPC-TRs/reports/CRPC-TR96640.pdf

    Annotations
    @Since( "1.3.0" )
    See also

    Isotonic regression (Wikipedia)

  4. class IsotonicRegressionModel extends Serializable with Saveable

    Permalink

    Regression model for isotonic regression.

    Regression model for isotonic regression.

    Annotations
    @Since( "1.3.0" )
  5. case class LabeledPoint(label: Double, features: Vector) extends Product with Serializable

    Permalink

    Class that represents the features and labels of a data point.

    Class that represents the features and labels of a data point.

    label

    Label for this data point.

    features

    List of features for this data point.

    Annotations
    @Since( "0.8.0" ) @BeanInfo()
  6. class LassoModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

    Permalink

    Regression model trained using Lasso.

    Regression model trained using Lasso.

    Annotations
    @Since( "0.8.0" )
  7. class LassoWithSGD extends GeneralizedLinearAlgorithm[LassoModel] with Serializable

    Permalink

    Train a regression model with L1-regularization using Stochastic Gradient Descent.

    Train a regression model with L1-regularization using Stochastic Gradient Descent. This solves the l1-regularized least squares regression formulation f(weights) = 1/2n ||A weights-y||2 + regParam ||weights||_1 Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

    Annotations
    @Since( "0.8.0" )
  8. class LinearRegressionModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

    Permalink

    Regression model trained using LinearRegression.

    Regression model trained using LinearRegression.

    Annotations
    @Since( "0.8.0" )
  9. class LinearRegressionWithSGD extends GeneralizedLinearAlgorithm[LinearRegressionModel] with Serializable

    Permalink

    Train a linear regression model with no regularization using Stochastic Gradient Descent.

    Train a linear regression model with no regularization using Stochastic Gradient Descent. This solves the least squares regression formulation f(weights) = 1/n ||A weights-y||2 (which is the mean squared error). Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

    Annotations
    @Since( "0.8.0" )
  10. trait RegressionModel extends Serializable

    Permalink
    Annotations
    @Since( "0.8.0" )
  11. class RidgeRegressionModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

    Permalink

    Regression model trained using RidgeRegression.

    Regression model trained using RidgeRegression.

    Annotations
    @Since( "0.8.0" )
  12. class RidgeRegressionWithSGD extends GeneralizedLinearAlgorithm[RidgeRegressionModel] with Serializable

    Permalink

    Train a regression model with L2-regularization using Stochastic Gradient Descent.

    Train a regression model with L2-regularization using Stochastic Gradient Descent. This solves the l2-regularized least squares regression formulation f(weights) = 1/2n ||A weights-y||2 + regParam/2 ||weights||2 Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

    Annotations
    @Since( "0.8.0" )
  13. abstract class StreamingLinearAlgorithm[M <: GeneralizedLinearModel, A <: GeneralizedLinearAlgorithm[M]] extends Logging

    Permalink

    :: DeveloperApi :: StreamingLinearAlgorithm implements methods for continuously training a generalized linear model model on streaming data, and using it for prediction on (possibly different) streaming data.

    :: DeveloperApi :: StreamingLinearAlgorithm implements methods for continuously training a generalized linear model model on streaming data, and using it for prediction on (possibly different) streaming data.

    This class takes as type parameters a GeneralizedLinearModel, and a GeneralizedLinearAlgorithm, making it easy to extend to construct streaming versions of any analyses using GLMs. Initial weights must be set before calling trainOn or predictOn. Only weights will be updated, not an intercept. If the model needs an intercept, it should be manually appended to the input data.

    For example usage, see StreamingLinearRegressionWithSGD.

    NOTE: In some use cases, the order in which trainOn and predictOn are called in an application will affect the results. When called on the same DStream, if trainOn is called before predictOn, when new data arrive the model will update and the prediction will be based on the new model. Whereas if predictOn is called first, the prediction will use the model from the previous update.

    NOTE: It is ok to call predictOn repeatedly on multiple streams; this will generate predictions for each one all using the current model. It is also ok to call trainOn on different streams; this will update the model using each of the different sources, in sequence.

    Annotations
    @Since( "1.1.0" ) @DeveloperApi()
  14. class StreamingLinearRegressionWithSGD extends StreamingLinearAlgorithm[LinearRegressionModel, LinearRegressionWithSGD] with Serializable

    Permalink

    Train or predict a linear regression model on streaming data.

    Train or predict a linear regression model on streaming data. Training uses Stochastic Gradient Descent to update the model based on each new batch of incoming data from a DStream (see LinearRegressionWithSGD for model equation)

    Each batch of data is assumed to be an RDD of LabeledPoints. The number of data points per batch can vary, but the number of features must be constant. An initial weight vector must be provided.

    Use a builder pattern to construct a streaming linear regression analysis in an application, like:

    val model = new StreamingLinearRegressionWithSGD() .setStepSize(0.5) .setNumIterations(10) .setInitialWeights(Vectors.dense(...)) .trainOn(DStream)

    Annotations
    @Since( "1.1.0" )

Value Members

  1. object IsotonicRegressionModel extends Loader[IsotonicRegressionModel] with Serializable

    Permalink
    Annotations
    @Since( "1.4.0" )
  2. object LabeledPoint extends Serializable

    Permalink

    Parser for org.apache.spark.mllib.regression.LabeledPoint.

    Annotations
    @Since( "1.1.0" )
  3. object LassoModel extends Loader[LassoModel] with Serializable

    Permalink
    Annotations
    @Since( "1.3.0" )
  4. object LinearRegressionModel extends Loader[LinearRegressionModel] with Serializable

    Permalink
    Annotations
    @Since( "1.3.0" )
  5. object RidgeRegressionModel extends Loader[RidgeRegressionModel] with Serializable

    Permalink
    Annotations
    @Since( "1.3.0" )

Deprecated Value Members

  1. object LassoWithSGD extends Serializable

    Permalink

    Top-level methods for calling Lasso.

    Top-level methods for calling Lasso.

    Annotations
    @Since( "0.8.0" ) @deprecated
    Deprecated

    (Since version 2.0.0)

  2. object LinearRegressionWithSGD extends Serializable

    Permalink

    Top-level methods for calling LinearRegression.

    Top-level methods for calling LinearRegression.

    Annotations
    @Since( "0.8.0" ) @deprecated
    Deprecated

    (Since version 2.0.0) Use ml.regression.LinearRegression or LBFGS

  3. object RidgeRegressionWithSGD extends Serializable

    Permalink

    Top-level methods for calling RidgeRegression.

    Top-level methods for calling RidgeRegression.

    Annotations
    @Since( "0.8.0" ) @deprecated
    Deprecated

    (Since version 2.0.0)

Ungrouped