PipelineModel#
- class pyspark.ml.connect.PipelineModel(stages=None)[source]#
Represents a compiled pipeline with transformers and fitted models.
New in version 3.5.0.
Methods
clear
(param)Clears a param from the param map if it has been explicitly set.
copy
([extra])Creates a copy of this instance.
explainParam
(param)Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
Returns the documentation of all params with their optionally default values and user-supplied values.
extractParamMap
([extra])Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
getOrDefault
(param)Gets the value of a param in the user-supplied param map or its default value.
getParam
(paramName)Gets a param by its name.
get_uid_map
(instance)hasDefault
(param)Checks whether a param has a default value.
hasParam
(paramName)Tests whether this instance contains a param with a given (string) name.
isDefined
(param)Checks whether a param is explicitly set by user or has a default value.
isSet
(param)Checks whether a param is explicitly set by user.
load
(path)Load Estimator / Transformer / Model / Evaluator from provided cloud storage path.
loadFromLocal
(path)Load Estimator / Transformer / Model / Evaluator from provided local path.
save
(path, *[, overwrite])Save Estimator / Transformer / Model / Evaluator to provided cloud storage path.
saveToLocal
(path, *[, overwrite])Save Estimator / Transformer / Model / Evaluator to provided local path.
set
(param, value)Sets a parameter in the embedded param map.
transform
(dataset[, params])Transforms the input dataset.
Attributes
Returns all params ordered by name.
Methods Documentation
- clear(param)#
Clears a param from the param map if it has been explicitly set.
- copy(extra=None)[source]#
Creates a copy of this instance.
New in version 3.5.0.
- Parameters
extra – extra parameters
- Returns
new instance
- explainParam(param)#
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
- explainParams()#
Returns the documentation of all params with their optionally default values and user-supplied values.
- extractParamMap(extra=None)#
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
- Parameters
- extradict, optional
extra param values
- Returns
- dict
merged param map
- getOrDefault(param)#
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
- getParam(paramName)#
Gets a param by its name.
- static get_uid_map(instance)#
- hasDefault(param)#
Checks whether a param has a default value.
- hasParam(paramName)#
Tests whether this instance contains a param with a given (string) name.
- isDefined(param)#
Checks whether a param is explicitly set by user or has a default value.
- isSet(param)#
Checks whether a param is explicitly set by user.
- classmethod load(path)#
Load Estimator / Transformer / Model / Evaluator from provided cloud storage path.
New in version 3.5.0.
- classmethod loadFromLocal(path)#
Load Estimator / Transformer / Model / Evaluator from provided local path.
New in version 3.5.0.
- save(path, *, overwrite=False)#
Save Estimator / Transformer / Model / Evaluator to provided cloud storage path.
New in version 3.5.0.
- saveToLocal(path, *, overwrite=False)#
Save Estimator / Transformer / Model / Evaluator to provided local path.
New in version 3.5.0.
- set(param, value)#
Sets a parameter in the embedded param map.
- transform(dataset, params=None)#
Transforms the input dataset. The dataset can be either pandas dataframe or spark dataframe, if it is a spark DataFrame, the result of transformation is a new spark DataFrame that contains all existing columns and output columns with names, If it is a pandas DataFrame, the result of transformation is a shallow copy of the input pandas dataframe with output columns with names.
Note: Transformers does not allow output column having the same name with existing columns.
- Parameters
- dataset
pyspark.sql.DataFrame
or py:class:pandas.DataFrame input dataset.
- paramsdict, optional
an optional param map that overrides embedded params.
- dataset
- Returns
pyspark.sql.DataFrame
or py:class:pandas.DataFrametransformed dataset, the type of output dataframe is consistent with input dataframe.
Attributes Documentation
- params#
Returns all params ordered by name. The default implementation uses
dir()
to get all attributes of typeParam
.
- uid#
A unique id for the object.