Type Members

1.  abstract class GeneralizedLinearAlgorithm[M <: GeneralizedLinearModel] extends Logging with Serializable

   GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM).

   GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM). This class should be extended with an Optimizer to create a new GLM.

   Annotations

   @Since("0.8.0")

2.  abstract class GeneralizedLinearModel extends Serializable

   GeneralizedLinearModel (GLM) represents a model trained using GeneralizedLinearAlgorithm.

   GeneralizedLinearModel (GLM) represents a model trained using GeneralizedLinearAlgorithm. GLMs consist of a weight vector and an intercept.

   Annotations

   @Since("0.8.0")

3.  class IsotonicRegression extends Serializable

   Isotonic regression.

   Isotonic regression. Currently implemented using parallelized pool adjacent violators algorithm. Only univariate (single feature) algorithm supported.

   Sequential PAV implementation based on: Grotzinger, S. J., and C. Witzgall. "Projections onto order simplexes." Applied mathematics and Optimization 12.1 (1984): 247-270.

   Sequential PAV parallelization based on: Kearsley, Anthony J., Richard A. Tapia, and Michael W. Trosset. "An approach to parallelizing isotonic regression." Applied Mathematics and Parallel Computing. Physica-Verlag HD, 1996. 141-147. Available from here

   Annotations

   @Since("1.3.0")

   See also

   Isotonic regression (Wikipedia)

4.  class IsotonicRegressionModel extends Serializable with Saveable

   Regression model for isotonic regression.

   Regression model for isotonic regression.

   Annotations

   @Since("1.3.0")

5.  case class LabeledPoint(label: Double, features: Vector) extends Product with Serializable

   Class that represents the features and labels of a data point.

   Class that represents the features and labels of a data point.

   label

   Label for this data point.

   features

   List of features for this data point.

   Annotations

   @Since("0.8.0")

6.  class LassoModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

   Regression model trained using Lasso.

   Regression model trained using Lasso.

   Annotations

   @Since("0.8.0")

7.  class LassoWithSGD extends GeneralizedLinearAlgorithm[LassoModel] with Serializable

   Train a regression model with L1-regularization using Stochastic Gradient Descent.

   Train a regression model with L1-regularization using Stochastic Gradient Descent. This solves the l1-regularized least squares regression formulation f(weights) = 1/2n ||A weights-y||² + regParam ||weights||_1 Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

   Annotations

   @Since("0.8.0")

8.  class LinearRegressionModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

   Regression model trained using LinearRegression.

   Regression model trained using LinearRegression.

   Annotations

   @Since("0.8.0")

9.  class LinearRegressionWithSGD extends GeneralizedLinearAlgorithm[LinearRegressionModel] with Serializable

   Train a linear regression model with no regularization using Stochastic Gradient Descent.

   Train a linear regression model with no regularization using Stochastic Gradient Descent. This solves the least squares regression formulation f(weights) = 1/n ||A weights-y||² (which is the mean squared error). Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

   Annotations

   @Since("0.8.0")

10.  trait RegressionModel extends Serializable

    Annotations

    @Since("0.8.0")

11.  class RidgeRegressionModel extends GeneralizedLinearModel with RegressionModel with Serializable with Saveable with PMMLExportable

    Regression model trained using RidgeRegression.

    Regression model trained using RidgeRegression.

    Annotations

    @Since("0.8.0")

12.  class RidgeRegressionWithSGD extends GeneralizedLinearAlgorithm[RidgeRegressionModel] with Serializable

    Train a regression model with L2-regularization using Stochastic Gradient Descent.

    Train a regression model with L2-regularization using Stochastic Gradient Descent. This solves the l2-regularized least squares regression formulation f(weights) = 1/2n ||A weights-y||² + regParam/2 ||weights||² Here the data matrix has n rows, and the input RDD holds the set of rows of A, each with its corresponding right hand side label y. See also the documentation for the precise formulation.

    Annotations

    @Since("0.8.0")

13.  abstract class StreamingLinearAlgorithm[M <: GeneralizedLinearModel, A <: GeneralizedLinearAlgorithm[M]] extends Logging

    StreamingLinearAlgorithm implements methods for continuously training a generalized linear model on streaming data, and using it for prediction on (possibly different) streaming data.

    StreamingLinearAlgorithm implements methods for continuously training a generalized linear model on streaming data, and using it for prediction on (possibly different) streaming data.

    This class takes as type parameters a GeneralizedLinearModel, and a GeneralizedLinearAlgorithm, making it easy to extend to construct streaming versions of any analyses using GLMs. Initial weights must be set before calling trainOn or predictOn. Only weights will be updated, not an intercept. If the model needs an intercept, it should be manually appended to the input data.

    For example usage, see StreamingLinearRegressionWithSGD.

    NOTE: In some use cases, the order in which trainOn and predictOn are called in an application will affect the results. When called on the same DStream, if trainOn is called before predictOn, when new data arrive the model will update and the prediction will be based on the new model. Whereas if predictOn is called first, the prediction will use the model from the previous update.

    NOTE: It is ok to call predictOn repeatedly on multiple streams; this will generate predictions for each one all using the current model. It is also ok to call trainOn on different streams; this will update the model using each of the different sources, in sequence.

    Annotations

    @Since("1.1.0")

14.  class StreamingLinearRegressionWithSGD extends StreamingLinearAlgorithm[LinearRegressionModel, LinearRegressionWithSGD] with Serializable

    Train or predict a linear regression model on streaming data.

    Train or predict a linear regression model on streaming data. Training uses Stochastic Gradient Descent to update the model based on each new batch of incoming data from a DStream (see LinearRegressionWithSGD for model equation)

    Each batch of data is assumed to be an RDD of LabeledPoints. The number of data points per batch can vary, but the number of features must be constant. An initial weight vector must be provided.

    Use a builder pattern to construct a streaming linear regression analysis in an application, like:

    val model = new StreamingLinearRegressionWithSGD() .setStepSize(0.5) .setNumIterations(10) .setInitialWeights(Vectors.dense(...)) .trainOn(DStream)

    Annotations

    @Since("1.1.0")