Package org.apache.spark.ml.feature

Class SQLTransformer

Object
org.apache.spark.ml.PipelineStage
org.apache.spark.ml.Transformer
org.apache.spark.ml.feature.SQLTransformer
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging, Params, DefaultParamsWritable, Identifiable, MLWritable, scala.Serializable
public class SQLTransformer extends Transformer implements DefaultParamsWritable
Implements the transformations which are defined by SQL statement. Currently we only support SQL syntax like 'SELECT ... FROM __THIS__ ...' where '__THIS__' represents the underlying table of the input dataset. The select clause specifies the fields, constants, and expressions to display in the output, it can be any select clause that Spark SQL supports. Users can also use Spark SQL built-in function and UDFs to operate on these selected columns. For example, SQLTransformer supports statements like: 

  SELECT a, a + b AS a_b FROM __THIS__
  SELECT a, SQRT(b) AS b_sqrt FROM __THIS__ where a > 5
  SELECT a, b, SUM(c) AS c_sum FROM __THIS__ GROUP BY a, b
See Also:

  • Constructor Details

    • SQLTransformer

      public SQLTransformer(String uid)

    • SQLTransformer

      public SQLTransformer()

  • Method Details

    • load

      public static SQLTransformer load(String path)

    • read

      public static MLReader<T> read()

    • uid

      public String uid()
      Description copied from interface: Identifiable
      An immutable unique ID for the object and its derivatives.
      Specified by:
      uid in interface Identifiable
      Returns:
      (undocumented)

    • statement

      public final Param<String> statement()
      SQL statement parameter. The statement is provided in string form.

      Returns:
      (undocumented)

    • setStatement

      public SQLTransformer setStatement(String value)

    • getStatement

      public String getStatement()

    • transform

      public Dataset<Row> transform(Dataset<?> dataset)
      Description copied from class: Transformer
      Transforms the input dataset.
      Specified by:
      transform in class Transformer
      Parameters:
      dataset - (undocumented)
      Returns:
      (undocumented)

    • transformSchema

      public StructType transformSchema(StructType schema)
      Description copied from class: PipelineStage
      Check transform validity and derive the output schema from the input schema.

      We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().

      Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.

      Specified by:
      transformSchema in class PipelineStage
      Parameters:
      schema - (undocumented)
      Returns:
      (undocumented)

    • copy

      public SQLTransformer copy(ParamMap extra)
      Description copied from interface: Params
      Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
      Specified by:
      copy in interface Params
      Specified by:
      copy in class Transformer
      Parameters:
      extra - (undocumented)
      Returns:
      (undocumented)

    • toString

      public String toString()
      Specified by:
      toString in interface Identifiable
      Overrides:
      toString in class Object