Package org.apache.spark.sql.connector.expressions

Class Expressions

Object
org.apache.spark.sql.connector.expressions.Expressions
@Evolving public class Expressions extends Object
Helper methods to create logical transforms to pass into Spark.
Since:
3.0.0

  • Method Details

    • apply

      public static Transform apply(String name, Expression... args)
      Create a logical transform for applying a named transform.

      This transform can represent applying any named transform.

      Parameters:
      name - the transform name
      args - expression arguments to the transform
      Returns:
      a logical transform

    • column

      public static NamedReference column(String name)
      Create a named reference expression for a (nested) column.
      Parameters:
      name - The column name. It refers to nested column if name contains dot.
      Returns:
      a named reference for the column

    • literal

      public static <T> Literal<T> literal(T value)
      Create a literal from a value.

      The JVM type of the value held by a literal must be the type used by Spark's InternalRow API for the literal's SQL data type.

      Type Parameters:
      T - the JVM type of the value
      Parameters:
      value - a value
      Returns:
      a literal expression for the value

    • bucket

      public static Transform bucket(int numBuckets, String... columns)
      Create a bucket transform for one or more columns.

      This transform represents a logical mapping from a value to a bucket id in [0, numBuckets) based on a hash of the value.

      The name reported by transforms created with this method is "bucket".

      Parameters:
      numBuckets - the number of output buckets
      columns - input columns for the bucket transform
      Returns:
      a logical bucket transform with name "bucket"

    • identity

      public static Transform identity(String column)
      Create an identity transform for a column.

      This transform represents a logical mapping from a value to itself.

      The name reported by transforms created with this method is "identity".

      Parameters:
      column - an input column
      Returns:
      a logical identity transform with name "identity"

    • years

      public static Transform years(String column)
      Create a yearly transform for a timestamp or date column.

      This transform represents a logical mapping from a timestamp or date to a year, such as 2018.

      The name reported by transforms created with this method is "years".

      Parameters:
      column - an input timestamp or date column
      Returns:
      a logical yearly transform with name "years"

    • months

      public static Transform months(String column)
      Create a monthly transform for a timestamp or date column.

      This transform represents a logical mapping from a timestamp or date to a month, such as 2018-05.

      The name reported by transforms created with this method is "months".

      Parameters:
      column - an input timestamp or date column
      Returns:
      a logical monthly transform with name "months"

    • days

      public static Transform days(String column)
      Create a daily transform for a timestamp or date column.

      This transform represents a logical mapping from a timestamp or date to a date, such as 2018-05-13.

      The name reported by transforms created with this method is "days".

      Parameters:
      column - an input timestamp or date column
      Returns:
      a logical daily transform with name "days"

    • hours

      public static Transform hours(String column)
      Create an hourly transform for a timestamp column.

      This transform represents a logical mapping from a timestamp to a date and hour, such as 2018-05-13, hour 19.

      The name reported by transforms created with this method is "hours".

      Parameters:
      column - an input timestamp column
      Returns:
      a logical hourly transform with name "hours"

    • sort

      public static SortOrder sort(Expression expr, SortDirection direction, NullOrdering nullOrder)
      Create a sort expression.
      Parameters:
      expr - an expression to produce values to sort
      direction - direction of the sort
      nullOrder - null order of the sort
      Returns:
      a SortOrder
      Since:
      3.2.0

    • sort

      public static SortOrder sort(Expression expr, SortDirection direction)
      Create a sort expression.
      Parameters:
      expr - an expression to produce values to sort
      direction - direction of the sort
      Returns:
      a SortOrder
      Since:
      3.2.0