Package org.apache.spark.ml.feature

Class RFormula

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Estimator<RFormulaModel>
org.apache.spark.ml.feature.RFormula
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, org.apache.spark.ml.feature.RFormulaBase, Params, HasFeaturesCol, HasHandleInvalid, HasLabelCol, DefaultParamsWritable, Identifiable, MLWritable
public class RFormula extends Estimator<RFormulaModel> implements org.apache.spark.ml.feature.RFormulaBase, DefaultParamsWritable
Implements the transforms required for fitting a dataset against an R model formula. Currently we support a limited subset of the R operators, including '~', '.', ':', '+', '-', '*' and '^'. Also see the R formula docs here: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/formula.html

The basic operators are: - ~ separate target and terms - + concat terms, "+ 0" means removing intercept - - remove a term, "- 1" means removing intercept - : interaction (multiplication for numeric values, or binarized categorical values) - . all columns except target - * factor crossing, includes the terms and interactions between them - ^ factor crossing to a specified degree

Suppose a and b are double columns, we use the following simple examples to illustrate the effect of RFormula: - y ~ a + b means model y ~ w0 + w1 * a + w2 * b where w0 is the intercept and w1, w2 are coefficients. - y ~ a + b + a:b - 1 means model y ~ w1 * a + w2 * b + w3 * a * b where w1, w2, w3 are coefficients. - y ~ a * b means model y ~ w0 + w1 * a + w2 * b + w3 * a * b where w0 is the intercept and w1, w2, w3 are coefficients - y ~ (a + b)^2 means model y ~ w0 + w1 * a + w2 * b + w3 * a * b where w0 is the intercept and w1, w2, w3 are coefficients

RFormula produces a vector column of features and a double or string column of label. Like when formulas are used in R for linear regression, string input columns will be one-hot encoded, and numeric columns will be cast to doubles. If the label column is of type string, it will be first transformed to double with StringIndexer. If the label column does not exist in the DataFrame, the output label column will be created from the specified response variable in the formula.

See Also:

  • Constructor Details

    • RFormula

      public RFormula(String uid)

    • RFormula

      public RFormula()

  • Method Details

    • load

      public static RFormula load(String path)

    • read

      public static MLReader<T> read()

    • formula

      public Param<String> formula()
      Specified by:
      formula in interface org.apache.spark.ml.feature.RFormulaBase

    • forceIndexLabel

      public BooleanParam forceIndexLabel()
      Specified by:
      forceIndexLabel in interface org.apache.spark.ml.feature.RFormulaBase

    • handleInvalid

      public Param<String> handleInvalid()
      Description copied from interface: HasHandleInvalid
      Param for how to handle invalid entries. Options are skip (which will filter out rows with bad values), or error (which will throw an error). More options may be added later.
      Specified by:
      handleInvalid in interface HasHandleInvalid
      Specified by:
      handleInvalid in interface org.apache.spark.ml.feature.RFormulaBase
      Returns:
      (undocumented)

    • stringIndexerOrderType

      public final Param<String> stringIndexerOrderType()
      Specified by:
      stringIndexerOrderType in interface org.apache.spark.ml.feature.RFormulaBase

    • labelCol

      public final Param<String> labelCol()
      Description copied from interface: HasLabelCol
      Param for label column name.
      Specified by:
      labelCol in interface HasLabelCol
      Returns:
      (undocumented)

    • featuresCol

      public final Param<String> featuresCol()
      Description copied from interface: HasFeaturesCol
      Param for features column name.
      Specified by:
      featuresCol in interface HasFeaturesCol
      Returns:
      (undocumented)

    • uid

      public String uid()
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Specified by:
      uid in interface Identifiable
      Returns:
      (undocumented)

    • setFormula

      public RFormula setFormula(String value)
      Sets the formula to use for this transformer. Must be called before use.
      Parameters:
      value - an R formula in string form (e.g. "y ~ x + z")
      Returns:
      (undocumented)

    • setHandleInvalid

      public RFormula setHandleInvalid(String value)

    • setFeaturesCol

      public RFormula setFeaturesCol(String value)

    • setLabelCol

      public RFormula setLabelCol(String value)

    • setForceIndexLabel

      public RFormula setForceIndexLabel(boolean value)

    • setStringIndexerOrderType

      public RFormula setStringIndexerOrderType(String value)

    • fit

      public RFormulaModel fit(Dataset<?> dataset)
      Description copied from class: Estimator
      Fits a model to the input data.
      Specified by:
      fit in class Estimator<RFormulaModel>
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Specified by:
      transformSchema in class PipelineStage
      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)

    • copy

      public RFormula copy(ParamMap extra)
      Description copied from interface: Params
      Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
      Specified by:
      copy in interface Params
      Specified by:
      copy in class Estimator<RFormulaModel>
      Parameters:
      extra - (undocumented)
      Returns:
      (undocumented)

    • toString

      public String toString()
      Specified by:
      toString in interface Identifiable
      Overrides:
      toString in class Object