Class GBTClassificationModel
- All Implemented Interfaces:
Serializable
,org.apache.spark.internal.Logging
,ClassifierParams
,ProbabilisticClassifierParams
,Params
,HasCheckpointInterval
,HasFeaturesCol
,HasLabelCol
,HasMaxIter
,HasPredictionCol
,HasProbabilityCol
,HasRawPredictionCol
,HasSeed
,HasStepSize
,HasThresholds
,HasValidationIndicatorCol
,HasWeightCol
,PredictorParams
,DecisionTreeParams
,GBTClassifierParams
,GBTParams
,HasVarianceImpurity
,TreeEnsembleClassifierParams
,TreeEnsembleModel<DecisionTreeRegressionModel>
,TreeEnsembleParams
,Identifiable
,MLWritable
param: _trees Decision trees in the ensemble. param: _treeWeights Weights for the decision trees in the ensemble.
- See Also:
- Note:
- Multiclass labels are not currently supported.
-
Nested Class Summary
Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging
org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter
-
Constructor Summary
ConstructorDescriptionGBTClassificationModel
(String uid, DecisionTreeRegressionModel[] _trees, double[] _treeWeights) Construct a GBTClassificationModel -
Method Summary
Modifier and TypeMethodDescriptionfinal BooleanParam
If false, the algorithm will pass trees to executors to match instances with nodes.final IntParam
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).Creates a copy of this instance with the same UID and some extra params.double[]
evaluateEachIteration
(Dataset<?> dataset) Method to compute error or loss for every iteration of gradient boosting.The number of features to consider for splits at each tree node.int
Number of trees in ensembleimpurity()
Criterion used for information gain calculation (case-insensitive).leafCol()
Leaf indices column name.static GBTClassificationModel
lossType()
Loss function which GBT tries to minimize.final IntParam
maxBins()
Maximum number of bins used for discretizing continuous features and for choosing how to split on features at each node.final IntParam
maxDepth()
Maximum depth of the tree (nonnegative).final IntParam
maxIter()
Param for maximum number of iterations (>= 0).final IntParam
Maximum memory in MB allocated to histogram aggregation.final DoubleParam
Minimum information gain for a split to be considered at a tree node.final IntParam
Minimum number of instances each child must have after split.final DoubleParam
Minimum fraction of the weighted sample count that each child must have after split.int
Number of classes (values which the label can take).int
Returns the number of features the model was trained on.double
Predict label for the given features.predictRaw
(Vector features) Raw prediction for each possible label.static MLReader<GBTClassificationModel>
read()
final LongParam
seed()
Param for random seed.final DoubleParam
stepSize()
Param for Step size (a.k.a.final DoubleParam
Fraction of the training data used for learning each decision tree, in range (0, 1].toString()
Summary of the modelint
Total number of nodes, summed over all trees in the ensemble.Transforms dataset by reading fromPredictionModel.featuresCol()
, and appending new columns as specified by parameters: - predicted labels asPredictionModel.predictionCol()
of typeDouble
- raw predictions (confidences) asClassificationModel.rawPredictionCol()
of typeVector
- probability of each class asProbabilisticClassificationModel.probabilityCol()
of typeVector
.transformSchema
(StructType schema) Check transform validity and derive the output schema from the input schema.trees()
Trees in this ensemble.double[]
Weights for each tree, zippable withTreeEnsembleModel.trees()
uid()
An immutable unique ID for the object and its derivatives.Param for name of the column that indicates whether each row is for training or for validation.final DoubleParam
Threshold for stopping early when fit with validation is used.Param for weight column name.write()
Returns anMLWriter
instance for this ML instance.Methods inherited from class org.apache.spark.ml.classification.ProbabilisticClassificationModel
normalizeToProbabilitiesInPlace, predictProbability, probabilityCol, setProbabilityCol, setThresholds, thresholds
Methods inherited from class org.apache.spark.ml.classification.ClassificationModel
rawPredictionCol, setRawPredictionCol, transformImpl
Methods inherited from class org.apache.spark.ml.PredictionModel
featuresCol, labelCol, predictionCol, setFeaturesCol, setPredictionCol
Methods inherited from class org.apache.spark.ml.Transformer
transform, transform, transform
Methods inherited from class org.apache.spark.ml.PipelineStage
params
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.apache.spark.ml.tree.DecisionTreeParams
getCacheNodeIds, getLeafCol, getMaxBins, getMaxDepth, getMaxMemoryInMB, getMinInfoGain, getMinInstancesPerNode, getMinWeightFractionPerNode, getOldStrategy, setLeafCol
Methods inherited from interface org.apache.spark.ml.tree.GBTClassifierParams
getLossType, getOldLossType
Methods inherited from interface org.apache.spark.ml.tree.GBTParams
getOldBoostingStrategy, getValidationTol
Methods inherited from interface org.apache.spark.ml.param.shared.HasCheckpointInterval
getCheckpointInterval
Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol
featuresCol, getFeaturesCol
Methods inherited from interface org.apache.spark.ml.param.shared.HasLabelCol
getLabelCol, labelCol
Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter
getMaxIter
Methods inherited from interface org.apache.spark.ml.param.shared.HasPredictionCol
getPredictionCol, predictionCol
Methods inherited from interface org.apache.spark.ml.param.shared.HasProbabilityCol
getProbabilityCol, probabilityCol
Methods inherited from interface org.apache.spark.ml.param.shared.HasRawPredictionCol
getRawPredictionCol, rawPredictionCol
Methods inherited from interface org.apache.spark.ml.param.shared.HasStepSize
getStepSize
Methods inherited from interface org.apache.spark.ml.param.shared.HasThresholds
getThresholds, thresholds
Methods inherited from interface org.apache.spark.ml.param.shared.HasValidationIndicatorCol
getValidationIndicatorCol
Methods inherited from interface org.apache.spark.ml.tree.HasVarianceImpurity
getImpurity, getOldImpurity
Methods inherited from interface org.apache.spark.ml.param.shared.HasWeightCol
getWeightCol
Methods inherited from interface org.apache.spark.internal.Logging
initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext
Methods inherited from interface org.apache.spark.ml.util.MLWritable
save
Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn
Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleClassifierParams
validateAndTransformSchema
Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleModel
getLeafField, javaTreeWeights, predictLeaf, toDebugString
Methods inherited from interface org.apache.spark.ml.tree.TreeEnsembleParams
getFeatureSubsetStrategy, getOldStrategy, getSubsamplingRate
-
Constructor Details
-
GBTClassificationModel
public GBTClassificationModel(String uid, DecisionTreeRegressionModel[] _trees, double[] _treeWeights) Construct a GBTClassificationModel- Parameters:
_trees
- Decision trees in the ensemble._treeWeights
- Weights for the decision trees in the ensemble.uid
- (undocumented)
-
-
Method Details
-
read
-
load
-
totalNumNodes
public int totalNumNodes()Description copied from interface:TreeEnsembleModel
Total number of nodes, summed over all trees in the ensemble.- Specified by:
totalNumNodes
in interfaceTreeEnsembleModel<DecisionTreeRegressionModel>
-
lossType
Description copied from interface:GBTClassifierParams
Loss function which GBT tries to minimize. (case-insensitive) Supported: "logistic" (default = logistic)- Specified by:
lossType
in interfaceGBTClassifierParams
- Returns:
- (undocumented)
-
impurity
Description copied from interface:HasVarianceImpurity
Criterion used for information gain calculation (case-insensitive). This impurity type is used in DecisionTreeRegressor, RandomForestRegressor, GBTRegressor and GBTClassifier (since GBTClassificationModel is internally composed of DecisionTreeRegressionModels). Supported: "variance". (default = variance)- Specified by:
impurity
in interfaceHasVarianceImpurity
- Returns:
- (undocumented)
-
validationTol
Description copied from interface:GBTParams
Threshold for stopping early when fit with validation is used. (This parameter is ignored when fit without validation is used.) The decision to stop early is decided based on this logic: If the current loss on the validation set is greater than 0.01, the diff of validation error is compared to relative tolerance which is validationTol * (current loss on the validation set). If the current loss on the validation set is less than or equal to 0.01, the diff of validation error is compared to absolute tolerance which is validationTol * 0.01.- Specified by:
validationTol
in interfaceGBTParams
- Returns:
- (undocumented)
- See Also:
-
stepSize
Description copied from interface:GBTParams
Param for Step size (a.k.a. learning rate) in interval (0, 1] for shrinking the contribution of each estimator. (default = 0.1)- Specified by:
stepSize
in interfaceGBTParams
- Specified by:
stepSize
in interfaceHasStepSize
- Returns:
- (undocumented)
-
validationIndicatorCol
Description copied from interface:HasValidationIndicatorCol
Param for name of the column that indicates whether each row is for training or for validation. False indicates training; true indicates validation..- Specified by:
validationIndicatorCol
in interfaceHasValidationIndicatorCol
- Returns:
- (undocumented)
-
maxIter
Description copied from interface:HasMaxIter
Param for maximum number of iterations (>= 0).- Specified by:
maxIter
in interfaceHasMaxIter
- Returns:
- (undocumented)
-
subsamplingRate
Description copied from interface:TreeEnsembleParams
Fraction of the training data used for learning each decision tree, in range (0, 1]. (default = 1.0)- Specified by:
subsamplingRate
in interfaceTreeEnsembleParams
- Returns:
- (undocumented)
-
featureSubsetStrategy
Description copied from interface:TreeEnsembleParams
The number of features to consider for splits at each tree node. Supported options: - "auto": Choose automatically for task: If numTrees == 1, set to "all." If numTrees greater than 1 (forest), set to "sqrt" for classification and to "onethird" for regression. - "all": use all features - "onethird": use 1/3 of the features - "sqrt": use sqrt(number of features) - "log2": use log2(number of features) - "n": when n is in the range (0, 1.0], use n * number of features. When n is in the range (1, number of features), use n features. (default = "auto")These various settings are based on the following references: - log2: tested in Breiman (2001) - sqrt: recommended by Breiman manual for random forests - The defaults of sqrt (classification) and onethird (regression) match the R randomForest package.
- Specified by:
featureSubsetStrategy
in interfaceTreeEnsembleParams
- Returns:
- (undocumented)
- See Also:
-
leafCol
Description copied from interface:DecisionTreeParams
Leaf indices column name. Predicted leaf index of each instance in each tree by preorder. (default = "")- Specified by:
leafCol
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
maxDepth
Description copied from interface:DecisionTreeParams
Maximum depth of the tree (nonnegative). E.g., depth 0 means 1 leaf node; depth 1 means 1 internal node + 2 leaf nodes. (default = 5)- Specified by:
maxDepth
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
maxBins
Description copied from interface:DecisionTreeParams
Maximum number of bins used for discretizing continuous features and for choosing how to split on features at each node. More bins give higher granularity. Must be at least 2 and at least number of categories in any categorical feature. (default = 32)- Specified by:
maxBins
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
minInstancesPerNode
Description copied from interface:DecisionTreeParams
Minimum number of instances each child must have after split. If a split causes the left or right child to have fewer than minInstancesPerNode, the split will be discarded as invalid. Must be at least 1. (default = 1)- Specified by:
minInstancesPerNode
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
minWeightFractionPerNode
Description copied from interface:DecisionTreeParams
Minimum fraction of the weighted sample count that each child must have after split. If a split causes the fraction of the total weight in the left or right child to be less than minWeightFractionPerNode, the split will be discarded as invalid. Should be in the interval [0.0, 0.5). (default = 0.0)- Specified by:
minWeightFractionPerNode
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
minInfoGain
Description copied from interface:DecisionTreeParams
Minimum information gain for a split to be considered at a tree node. Should be at least 0.0. (default = 0.0)- Specified by:
minInfoGain
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
maxMemoryInMB
Description copied from interface:DecisionTreeParams
Maximum memory in MB allocated to histogram aggregation. If too small, then 1 node will be split per iteration, and its aggregates may exceed this size. (default = 256 MB)- Specified by:
maxMemoryInMB
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
cacheNodeIds
Description copied from interface:DecisionTreeParams
If false, the algorithm will pass trees to executors to match instances with nodes. If true, the algorithm will cache node IDs for each instance. Caching can speed up training of deeper trees. Users can set how often should the cache be checkpointed or disable it by setting checkpointInterval. (default = false)- Specified by:
cacheNodeIds
in interfaceDecisionTreeParams
- Returns:
- (undocumented)
-
weightCol
Description copied from interface:HasWeightCol
Param for weight column name. If this is not set or empty, we treat all instance weights as 1.0.- Specified by:
weightCol
in interfaceHasWeightCol
- Returns:
- (undocumented)
-
seed
Description copied from interface:HasSeed
Param for random seed. -
checkpointInterval
Description copied from interface:HasCheckpointInterval
Param for set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.- Specified by:
checkpointInterval
in interfaceHasCheckpointInterval
- Returns:
- (undocumented)
-
uid
Description copied from interface:Identifiable
An immutable unique ID for the object and its derivatives.- Specified by:
uid
in interfaceIdentifiable
- Returns:
- (undocumented)
-
numFeatures
public int numFeatures()Description copied from class:PredictionModel
Returns the number of features the model was trained on. If unknown, returns -1- Overrides:
numFeatures
in classPredictionModel<Vector,
GBTClassificationModel>
-
numClasses
public int numClasses()Description copied from class:ClassificationModel
Number of classes (values which the label can take).- Specified by:
numClasses
in classClassificationModel<Vector,
GBTClassificationModel>
-
trees
Description copied from interface:TreeEnsembleModel
Trees in this ensemble. Warning: These have null parent Estimators.- Specified by:
trees
in interfaceTreeEnsembleModel<DecisionTreeRegressionModel>
-
getNumTrees
public int getNumTrees()Number of trees in ensemble- Returns:
- (undocumented)
-
treeWeights
public double[] treeWeights()Description copied from interface:TreeEnsembleModel
Weights for each tree, zippable withTreeEnsembleModel.trees()
- Specified by:
treeWeights
in interfaceTreeEnsembleModel<DecisionTreeRegressionModel>
-
transformSchema
Description copied from class:PipelineStage
Check transform validity and derive the output schema from the input schema.We check validity for interactions between parameters during
transformSchema
and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled byParam.validate()
.Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
- Overrides:
transformSchema
in classProbabilisticClassificationModel<Vector,
GBTClassificationModel> - Parameters:
schema
- (undocumented)- Returns:
- (undocumented)
-
transform
Description copied from class:ProbabilisticClassificationModel
Transforms dataset by reading fromPredictionModel.featuresCol()
, and appending new columns as specified by parameters: - predicted labels asPredictionModel.predictionCol()
of typeDouble
- raw predictions (confidences) asClassificationModel.rawPredictionCol()
of typeVector
- probability of each class asProbabilisticClassificationModel.probabilityCol()
of typeVector
.- Overrides:
transform
in classProbabilisticClassificationModel<Vector,
GBTClassificationModel> - Parameters:
dataset
- input dataset- Returns:
- transformed dataset
-
predict
Description copied from class:ClassificationModel
Predict label for the given features. This method is used to implementtransform()
and outputPredictionModel.predictionCol()
.This default implementation for classification predicts the index of the maximum value from
predictRaw()
.- Overrides:
predict
in classClassificationModel<Vector,
GBTClassificationModel> - Parameters:
features
- (undocumented)- Returns:
- (undocumented)
-
predictRaw
Description copied from class:ClassificationModel
Raw prediction for each possible label. The meaning of a "raw" prediction may vary between algorithms, but it intuitively gives a measure of confidence in each possible label (where larger = more confident). This internal method is used to implementtransform()
and outputClassificationModel.rawPredictionCol()
.- Specified by:
predictRaw
in classClassificationModel<Vector,
GBTClassificationModel> - Parameters:
features
- (undocumented)- Returns:
- vector where element i is the raw prediction for label i. This raw prediction may be any real number, where a larger value indicates greater confidence for that label.
-
copy
Description copied from interface:Params
Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. SeedefaultCopy()
.- Specified by:
copy
in interfaceParams
- Specified by:
copy
in classModel<GBTClassificationModel>
- Parameters:
extra
- (undocumented)- Returns:
- (undocumented)
-
toString
Description copied from interface:TreeEnsembleModel
Summary of the model- Specified by:
toString
in interfaceIdentifiable
- Specified by:
toString
in interfaceTreeEnsembleModel<DecisionTreeRegressionModel>
- Overrides:
toString
in classObject
-
featureImportances
-
evaluateEachIteration
Method to compute error or loss for every iteration of gradient boosting.- Parameters:
dataset
- Dataset for validation.- Returns:
- (undocumented)
-
write
Description copied from interface:MLWritable
Returns anMLWriter
instance for this ML instance.- Specified by:
write
in interfaceMLWritable
- Returns:
- (undocumented)
-