Package org.apache.spark.ml
Class Pipeline
- All Implemented Interfaces:
Serializable
,org.apache.spark.internal.Logging
,Params
,Identifiable
,MLWritable
A simple pipeline, which acts as an estimator. A Pipeline consists of a sequence of stages, each
of which is either an
Estimator
or a Transformer
. When Pipeline.fit
is called, the
stages are executed in order. If a stage is an Estimator
, its Estimator.fit
method will
be called on the input dataset to fit a model. Then the model, which is a transformer, will be
used to transform the dataset as the input to the next stage. If a stage is a Transformer
,
its Transformer.transform
method will be called to produce the dataset for the next stage.
The fitted model from a Pipeline
is a PipelineModel
, which consists of fitted models and
transformers, corresponding to the pipeline stages. If there are no stages, the pipeline acts as
an identity transformer.- See Also:
-
Nested Class Summary
Modifier and TypeClassDescriptionstatic class
Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging
org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionCreates a copy of this instance with the same UID and some extra params.Fits the pipeline to the input dataset with additional parameters.static Pipeline
read()
setStages
(PipelineStage[] value) stages()
param for pipeline stagestransformSchema
(StructType schema) Check transform validity and derive the output schema from the input schema.uid()
An immutable unique ID for the object and its derivatives.write()
Returns anMLWriter
instance for this ML instance.Methods inherited from class org.apache.spark.ml.PipelineStage
params
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString
Methods inherited from interface org.apache.spark.internal.Logging
initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext
Methods inherited from interface org.apache.spark.ml.util.MLWritable
save
Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, onParamChange, set, set, set, setDefault, setDefault, shouldOwn
-
Constructor Details
-
Pipeline
-
Pipeline
public Pipeline()
-
-
Method Details
-
read
-
load
-
uid
Description copied from interface:Identifiable
An immutable unique ID for the object and its derivatives.- Specified by:
uid
in interfaceIdentifiable
- Returns:
- (undocumented)
-
stages
param for pipeline stages- Returns:
- (undocumented)
-
setStages
-
getStages
-
fit
Fits the pipeline to the input dataset with additional parameters. If a stage is anEstimator
, itsEstimator.fit
method will be called on the input dataset to fit a model. Then the model, which is a transformer, will be used to transform the dataset as the input to the next stage. If a stage is aTransformer
, itsTransformer.transform
method will be called to produce the dataset for the next stage. The fitted model from aPipeline
is anPipelineModel
, which consists of fitted models and transformers, corresponding to the pipeline stages. If there are no stages, the output model acts as an identity transformer.- Specified by:
fit
in classEstimator<PipelineModel>
- Parameters:
dataset
- input dataset- Returns:
- fitted pipeline
-
copy
Description copied from interface:Params
Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. SeedefaultCopy()
.- Specified by:
copy
in interfaceParams
- Specified by:
copy
in classEstimator<PipelineModel>
- Parameters:
extra
- (undocumented)- Returns:
- (undocumented)
-
transformSchema
Description copied from class:PipelineStage
Check transform validity and derive the output schema from the input schema.We check validity for interactions between parameters during
transformSchema
and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled byParam.validate()
.Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
- Specified by:
transformSchema
in classPipelineStage
- Parameters:
schema
- (undocumented)- Returns:
- (undocumented)
-
write
Description copied from interface:MLWritable
Returns anMLWriter
instance for this ML instance.- Specified by:
write
in interfaceMLWritable
- Returns:
- (undocumented)
-