Package org.apache.spark.ml.regression

Class GBTRegressionModel

Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Transformer

org.apache.spark.ml.Model<M>

org.apache.spark.ml.PredictionModel<FeaturesType,M>

org.apache.spark.ml.regression.RegressionModel<Vector,GBTRegressionModel>

org.apache.spark.ml.regression.GBTRegressionModel

All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging, Params, HasCheckpointInterval, HasFeaturesCol, HasLabelCol, HasMaxIter, HasPredictionCol, HasSeed, HasStepSize, HasValidationIndicatorCol, HasWeightCol, org.apache.spark.ml.PredictorParams, org.apache.spark.ml.tree.DecisionTreeParams, org.apache.spark.ml.tree.GBTParams, org.apache.spark.ml.tree.GBTRegressorParams, org.apache.spark.ml.tree.HasVarianceImpurity, org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>, org.apache.spark.ml.tree.TreeEnsembleParams, org.apache.spark.ml.tree.TreeEnsembleRegressorParams, org.apache.spark.ml.tree.TreeRegressorParams, Identifiable, MLWritable

public class GBTRegressionModel
extends RegressionModel<Vector,GBTRegressionModel>
implements org.apache.spark.ml.tree.GBTRegressorParams, org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>, MLWritable, Serializable

Gradient-Boosted Trees (GBTs) model for regression. It supports both continuous and categorical features. param: _trees Decision trees in the ensemble. param: _treeWeights Weights for the decision trees in the ensemble.

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Constructor Summary

Constructors

Constructor	Description
GBTRegressionModel(String uid, DecisionTreeRegressionModel[] _trees, double[] _treeWeights)	Construct a GBTRegressionModel

Method Summary

Modifier and Type	Method	Description
final BooleanParam	cacheNodeIds()
final IntParam	checkpointInterval()	Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
GBTRegressionModel	copy(ParamMap extra)	Creates a copy of this instance with the same UID and some extra params.
double[]	evaluateEachIteration(Dataset<?> dataset, String loss)	Method to compute error or loss for every iteration of gradient boosting.
Vector	featureImportances()
final Param<String>	featureSubsetStrategy()
int	getNumTrees()	Number of trees in ensemble
final Param<String>	impurity()
final Param<String>	leafCol()
static GBTRegressionModel	load(String path)
Param<String>	lossType()
final IntParam	maxBins()
final IntParam	maxDepth()
final IntParam	maxIter()	Param for maximum number of iterations (&gt;= 0).
final IntParam	maxMemoryInMB()
final DoubleParam	minInfoGain()
final IntParam	minInstancesPerNode()
final DoubleParam	minWeightFractionPerNode()
int	numFeatures()	Returns the number of features the model was trained on.
double	predict(Vector features)	Predict label for the given features.
static MLReader<GBTRegressionModel>	read()
final LongParam	seed()	Param for random seed.
final DoubleParam	stepSize()	Param for Step size to be used for each iteration of optimization (&gt; 0).
final DoubleParam	subsamplingRate()
String	toString()
int	totalNumNodes()
Dataset<Row>	transform(Dataset<?> dataset)	Transforms dataset by reading from PredictionModel.featuresCol(), calling predict, and storing the predictions as a new column PredictionModel.predictionCol().
StructType	transformSchema(StructType schema)	Check transform validity and derive the output schema from the input schema.
DecisionTreeRegressionModel[]	trees()
double[]	treeWeights()
String	uid()	An immutable unique ID for the object and its derivatives.
final Param<String>	validationIndicatorCol()	Param for name of the column that indicates whether each row is for training or for validation.
final DoubleParam	validationTol()
final Param<String>	weightCol()	Param for weight column name.
MLWriter	write()	Returns an MLWriter instance for this ML instance.

Methods inherited from class org.apache.spark.ml.PredictionModel

featuresCol, labelCol, predictionCol, setFeaturesCol, setPredictionCol

Methods inherited from class org.apache.spark.ml.Model

hasParent, parent, setParent

Methods inherited from class org.apache.spark.ml.Transformer

transform, transform, transform

Methods inherited from class org.apache.spark.ml.PipelineStage

params

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.tree.DecisionTreeParams

getCacheNodeIds, getLeafCol, getMaxBins, getMaxDepth, getMaxMemoryInMB, getMinInfoGain, getMinInstancesPerNode, getMinWeightFractionPerNode, getOldStrategy, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$cacheNodeIds_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$leafCol_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$maxBins_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$maxDepth_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$maxMemoryInMB_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$minInfoGain_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$minInstancesPerNode_$eq, org$apache$spark$ml$tree$DecisionTreeParams$_setter_$minWeightFractionPerNode_$eq, setLeafCol

Methods inherited from interface org.apache.spark.ml.tree.GBTParams

getOldBoostingStrategy, getValidationTol, org$apache$spark$ml$tree$GBTParams$_setter_$stepSize_$eq, org$apache$spark$ml$tree$GBTParams$_setter_$validationTol_$eq

Methods inherited from interface org.apache.spark.ml.tree.GBTRegressorParams

convertToOldLossType, getLossType, getOldLossType, org$apache$spark$ml$tree$GBTRegressorParams$_setter_$lossType_$eq

Methods inherited from interface org.apache.spark.ml.param.shared.HasCheckpointInterval

getCheckpointInterval

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol

featuresCol, getFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasLabelCol

getLabelCol, labelCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter

getMaxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasPredictionCol

getPredictionCol, predictionCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed

getSeed

Methods inherited from interface org.apache.spark.ml.param.shared.HasStepSize

getStepSize

Methods inherited from interface org.apache.spark.ml.param.shared.HasValidationIndicatorCol

getValidationIndicatorCol

Methods inherited from interface org.apache.spark.ml.tree.HasVarianceImpurity

getImpurity, getOldImpurity, org$apache$spark$ml$tree$HasVarianceImpurity$_setter_$impurity_$eq

Methods inherited from interface org.apache.spark.ml.param.shared.HasWeightCol

getWeightCol

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Methods inherited from interface org.apache.spark.ml.util.MLWritable

save

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, estimateMatadataSize, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleModel

getEstimatedSize, getLeafField, getTree, javaTreeWeights, predictLeaf, toDebugString

Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleParams

getFeatureSubsetStrategy, getOldStrategy, getSubsamplingRate, org$apache$spark$ml$tree$TreeEnsembleParams$_setter_$featureSubsetStrategy_$eq, org$apache$spark$ml$tree$TreeEnsembleParams$_setter_$subsamplingRate_$eq

Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleRegressorParams

validateAndTransformSchema

Constructor Details

GBTRegressionModel

public GBTRegressionModel(String uid,
 DecisionTreeRegressionModel[] _trees,
 double[] _treeWeights)

Construct a GBTRegressionModel

Parameters:

_trees - Decision trees in the ensemble.

_treeWeights - Weights for the decision trees in the ensemble.

uid - (undocumented)

Method Details

read

public static MLReader<GBTRegressionModel> read()

load

public static GBTRegressionModel load(String path)

totalNumNodes

public int totalNumNodes()

Specified by:

totalNumNodes in interface org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>

lossType

public Param<String> lossType()

Specified by:

lossType in interface org.apache.spark.ml.tree.GBTRegressorParams

impurity

public final Param<String> impurity()

Specified by:

impurity in interface org.apache.spark.ml.tree.HasVarianceImpurity

validationTol

public final DoubleParam validationTol()

Specified by:

validationTol in interface org.apache.spark.ml.tree.GBTParams

stepSize

public final DoubleParam stepSize()

Description copied from interface: HasStepSize

Param for Step size to be used for each iteration of optimization (&gt; 0).

Specified by:

stepSize in interface org.apache.spark.ml.tree.GBTParams

Specified by:

stepSize in interface HasStepSize

Returns:

(undocumented)

validationIndicatorCol

public final Param<String> validationIndicatorCol()

Description copied from interface: HasValidationIndicatorCol

Param for name of the column that indicates whether each row is for training or for validation. False indicates training; true indicates validation..

Specified by:

validationIndicatorCol in interface HasValidationIndicatorCol

Returns:

(undocumented)

maxIter

public final IntParam maxIter()

Description copied from interface: HasMaxIter

Param for maximum number of iterations (&gt;= 0).

Specified by:

maxIter in interface HasMaxIter

Returns:

(undocumented)

subsamplingRate

public final DoubleParam subsamplingRate()

Specified by:

subsamplingRate in interface org.apache.spark.ml.tree.TreeEnsembleParams

featureSubsetStrategy

public final Param<String> featureSubsetStrategy()

Specified by:

featureSubsetStrategy in interface org.apache.spark.ml.tree.TreeEnsembleParams

leafCol

public final Param<String> leafCol()

Specified by:

leafCol in interface org.apache.spark.ml.tree.DecisionTreeParams

maxDepth

public final IntParam maxDepth()

Specified by:

maxDepth in interface org.apache.spark.ml.tree.DecisionTreeParams

maxBins

public final IntParam maxBins()

Specified by:

maxBins in interface org.apache.spark.ml.tree.DecisionTreeParams

minInstancesPerNode

public final IntParam minInstancesPerNode()

Specified by:

minInstancesPerNode in interface org.apache.spark.ml.tree.DecisionTreeParams

minWeightFractionPerNode

public final DoubleParam minWeightFractionPerNode()

Specified by:

minWeightFractionPerNode in interface org.apache.spark.ml.tree.DecisionTreeParams

minInfoGain

public final DoubleParam minInfoGain()

Specified by:

minInfoGain in interface org.apache.spark.ml.tree.DecisionTreeParams

maxMemoryInMB

public final IntParam maxMemoryInMB()

Specified by:

maxMemoryInMB in interface org.apache.spark.ml.tree.DecisionTreeParams

cacheNodeIds

public final BooleanParam cacheNodeIds()

Specified by:

cacheNodeIds in interface org.apache.spark.ml.tree.DecisionTreeParams

weightCol

public final Param<String> weightCol()

Description copied from interface: HasWeightCol

Param for weight column name. If this is not set or empty, we treat all instance weights as 1.0.

Specified by:

weightCol in interface HasWeightCol

Returns:

(undocumented)

seed

public final LongParam seed()

Description copied from interface: HasSeed

Param for random seed.

Specified by:

seed in interface HasSeed

Returns:

(undocumented)

checkpointInterval

public final IntParam checkpointInterval()

Description copied from interface: HasCheckpointInterval

Param for set checkpoint interval (&gt;= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.

Specified by:

checkpointInterval in interface HasCheckpointInterval

Returns:

(undocumented)

uid

public String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

numFeatures

public int numFeatures()

Description copied from class: PredictionModel

Returns the number of features the model was trained on. If unknown, returns -1

Overrides:

numFeatures in class PredictionModel<Vector,GBTRegressionModel>

trees

public DecisionTreeRegressionModel[] trees()

Specified by:

trees in interface org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>

getNumTrees

public int getNumTrees()

Number of trees in ensemble

Returns:

(undocumented)

treeWeights

public double[] treeWeights()

Specified by:

treeWeights in interface org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

Check transform validity and derive the output schema from the input schema.

We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

Overrides:

transformSchema in class PredictionModel<Vector,GBTRegressionModel>

Parameters:

schema - (undocumented)

Returns:

(undocumented)

transform

public Dataset<Row> transform(Dataset<?> dataset)

Description copied from class: PredictionModel

Transforms dataset by reading from PredictionModel.featuresCol(), calling predict, and storing the predictions as a new column PredictionModel.predictionCol().

Overrides:

transform in class PredictionModel<Vector,GBTRegressionModel>

Parameters:

dataset - input dataset

Returns:

transformed dataset with PredictionModel.predictionCol() of type Double

predict

public double predict(Vector features)

Description copied from class: PredictionModel

Predict label for the given features. This method is used to implement transform() and output PredictionModel.predictionCol().

Specified by:

predict in class PredictionModel<Vector,GBTRegressionModel>

Parameters:

features - (undocumented)

Returns:

(undocumented)

copy

public GBTRegressionModel copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().

Specified by:

copy in interface Params

Specified by:

copy in class Model<GBTRegressionModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

toString

public String toString()

Specified by:

toString in interface Identifiable

Specified by:

toString in interface org.apache.spark.ml.tree.TreeEnsembleModel<DecisionTreeRegressionModel>

Overrides:

toString in class Object

featureImportances

public Vector featureImportances()

evaluateEachIteration

public double[] evaluateEachIteration(Dataset<?> dataset,
 String loss)

Method to compute error or loss for every iteration of gradient boosting.

Parameters:

dataset - Dataset for validation.

loss - The loss function used to compute error. Supported options: squared, absolute

Returns:

(undocumented)

write

public MLWriter write()

Description copied from interface: MLWritable

Returns an MLWriter instance for this ML instance.

Specified by:

write in interface MLWritable

Returns:

(undocumented)