org.apache.spark.sql

functions

object functions

:: Experimental :: Functions available for DataFrame.

Annotations
@Experimental()
Linear Supertypes
AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By inheritance
Inherited
  1. functions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def abs(e: Column): Column

    Computes the absolutle value.

  7. def approxCountDistinct(columnName: String, rsd: Double): Column

    Aggregate function: returns the approximate number of distinct items in a group.

  8. def approxCountDistinct(e: Column, rsd: Double): Column

    Aggregate function: returns the approximate number of distinct items in a group.

  9. def approxCountDistinct(columnName: String): Column

    Aggregate function: returns the approximate number of distinct items in a group.

  10. def approxCountDistinct(e: Column): Column

    Aggregate function: returns the approximate number of distinct items in a group.

  11. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  12. def asc(columnName: String): Column

    Returns a sort expression based on ascending order of the column.

    Returns a sort expression based on ascending order of the column. {{ // Sort by dept in ascending order, and then age in descending order. df.sort(asc("dept"), desc("age")) }}

  13. def avg(columnName: String): Column

    Aggregate function: returns the average of the values in a group.

  14. def avg(e: Column): Column

    Aggregate function: returns the average of the values in a group.

  15. def callUDF(f: Function10[_, _, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column, arg10: Column): Column

    Call a Scala function of 10 arguments as user-defined function (UDF).

    Call a Scala function of 10 arguments as user-defined function (UDF). This requires you to specify the return data type.

  16. def callUDF(f: Function9[_, _, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column, arg9: Column): Column

    Call a Scala function of 9 arguments as user-defined function (UDF).

    Call a Scala function of 9 arguments as user-defined function (UDF). This requires you to specify the return data type.

  17. def callUDF(f: Function8[_, _, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column, arg8: Column): Column

    Call a Scala function of 8 arguments as user-defined function (UDF).

    Call a Scala function of 8 arguments as user-defined function (UDF). This requires you to specify the return data type.

  18. def callUDF(f: Function7[_, _, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column, arg7: Column): Column

    Call a Scala function of 7 arguments as user-defined function (UDF).

    Call a Scala function of 7 arguments as user-defined function (UDF). This requires you to specify the return data type.

  19. def callUDF(f: Function6[_, _, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column, arg6: Column): Column

    Call a Scala function of 6 arguments as user-defined function (UDF).

    Call a Scala function of 6 arguments as user-defined function (UDF). This requires you to specify the return data type.

  20. def callUDF(f: Function5[_, _, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column, arg5: Column): Column

    Call a Scala function of 5 arguments as user-defined function (UDF).

    Call a Scala function of 5 arguments as user-defined function (UDF). This requires you to specify the return data type.

  21. def callUDF(f: Function4[_, _, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column, arg4: Column): Column

    Call a Scala function of 4 arguments as user-defined function (UDF).

    Call a Scala function of 4 arguments as user-defined function (UDF). This requires you to specify the return data type.

  22. def callUDF(f: Function3[_, _, _, _], returnType: DataType, arg1: Column, arg2: Column, arg3: Column): Column

    Call a Scala function of 3 arguments as user-defined function (UDF).

    Call a Scala function of 3 arguments as user-defined function (UDF). This requires you to specify the return data type.

  23. def callUDF(f: Function2[_, _, _], returnType: DataType, arg1: Column, arg2: Column): Column

    Call a Scala function of 2 arguments as user-defined function (UDF).

    Call a Scala function of 2 arguments as user-defined function (UDF). This requires you to specify the return data type.

  24. def callUDF(f: Function1[_, _], returnType: DataType, arg1: Column): Column

    Call a Scala function of 1 arguments as user-defined function (UDF).

    Call a Scala function of 1 arguments as user-defined function (UDF). This requires you to specify the return data type.

  25. def callUDF(f: Function0[_], returnType: DataType): Column

    Call a Scala function of 0 arguments as user-defined function (UDF).

    Call a Scala function of 0 arguments as user-defined function (UDF). This requires you to specify the return data type.

  26. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  27. def coalesce(e: Column*): Column

    Returns the first column that is not null.

    Returns the first column that is not null.

    df.select(coalesce(df("a"), df("b")))
    Annotations
    @varargs()
  28. def col(colName: String): Column

    Returns a Column based on the given column name.

  29. def column(colName: String): Column

    Returns a Column based on the given column name.

    Returns a Column based on the given column name. Alias of col.

  30. def count(columnName: String): Column

    Aggregate function: returns the number of items in a group.

  31. def count(e: Column): Column

    Aggregate function: returns the number of items in a group.

  32. def countDistinct(columnName: String, columnNames: String*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations
    @varargs()
  33. def countDistinct(expr: Column, exprs: Column*): Column

    Aggregate function: returns the number of distinct items in a group.

    Aggregate function: returns the number of distinct items in a group.

    Annotations
    @varargs()
  34. def desc(columnName: String): Column

    Returns a sort expression based on the descending order of the column.

    Returns a sort expression based on the descending order of the column. {{ // Sort by dept in ascending order, and then age in descending order. df.sort(asc("dept"), desc("age")) }}

  35. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  36. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  37. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  38. def first(columnName: String): Column

    Aggregate function: returns the first value of a column in a group.

  39. def first(e: Column): Column

    Aggregate function: returns the first value in a group.

  40. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  41. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  42. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  43. def last(columnName: String): Column

    Aggregate function: returns the last value of the column in a group.

  44. def last(e: Column): Column

    Aggregate function: returns the last value in a group.

  45. def lit(literal: Any): Column

    Creates a Column of literal value.

    Creates a Column of literal value.

    The passed in object is returned directly if it is already a Column. If the object is a Scala Symbol, it is converted into a Column also. Otherwise, a new Column is created to represent the literal value.

  46. def lower(e: Column): Column

    Converts a string exprsesion to lower case.

  47. def max(columnName: String): Column

    Aggregate function: returns the maximum value of the column in a group.

  48. def max(e: Column): Column

    Aggregate function: returns the maximum value of the expression in a group.

  49. def min(columnName: String): Column

    Aggregate function: returns the minimum value of the column in a group.

  50. def min(e: Column): Column

    Aggregate function: returns the minimum value of the expression in a group.

  51. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  52. def negate(e: Column): Column

    Unary minus, i.

    Unary minus, i.e. negate the expression.

    // Select the amount column and negates all values.
    // Scala:
    df.select( -df("amount") )
    
    // Java:
    df.select( negate(df.col("amount")) );
  53. def not(e: Column): Column

    Inversion of boolean expression, i.

    Inversion of boolean expression, i.e. NOT. {{ // Scala: select rows that are not active (isActive === false) df.filter( !df("isActive") )

    // Java: df.filter( not(df.col("isActive")) ); }}

  54. final def notify(): Unit

    Definition Classes
    AnyRef
  55. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  56. def sqrt(e: Column): Column

    Computes the square root of the specified float value.

  57. def sum(columnName: String): Column

    Aggregate function: returns the sum of all values in the given column.

  58. def sum(e: Column): Column

    Aggregate function: returns the sum of all values in the expression.

  59. def sumDistinct(columnName: String): Column

    Aggregate function: returns the sum of distinct values in the expression.

  60. def sumDistinct(e: Column): Column

    Aggregate function: returns the sum of distinct values in the expression.

  61. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  62. def toString(): String

    Definition Classes
    AnyRef → Any
  63. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9, A10](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9, A10) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9], arg10: scala.reflect.api.JavaUniverse.TypeTag[A10]): UserDefinedFunction

    Defines a user-defined function of 10 arguments as user-defined function (UDF).

    Defines a user-defined function of 10 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  64. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8, A9](f: (A1, A2, A3, A4, A5, A6, A7, A8, A9) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8], arg9: scala.reflect.api.JavaUniverse.TypeTag[A9]): UserDefinedFunction

    Defines a user-defined function of 9 arguments as user-defined function (UDF).

    Defines a user-defined function of 9 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  65. def udf[RT, A1, A2, A3, A4, A5, A6, A7, A8](f: (A1, A2, A3, A4, A5, A6, A7, A8) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7], arg8: scala.reflect.api.JavaUniverse.TypeTag[A8]): UserDefinedFunction

    Defines a user-defined function of 8 arguments as user-defined function (UDF).

    Defines a user-defined function of 8 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  66. def udf[RT, A1, A2, A3, A4, A5, A6, A7](f: (A1, A2, A3, A4, A5, A6, A7) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6], arg7: scala.reflect.api.JavaUniverse.TypeTag[A7]): UserDefinedFunction

    Defines a user-defined function of 7 arguments as user-defined function (UDF).

    Defines a user-defined function of 7 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  67. def udf[RT, A1, A2, A3, A4, A5, A6](f: (A1, A2, A3, A4, A5, A6) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5], arg6: scala.reflect.api.JavaUniverse.TypeTag[A6]): UserDefinedFunction

    Defines a user-defined function of 6 arguments as user-defined function (UDF).

    Defines a user-defined function of 6 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  68. def udf[RT, A1, A2, A3, A4, A5](f: (A1, A2, A3, A4, A5) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4], arg5: scala.reflect.api.JavaUniverse.TypeTag[A5]): UserDefinedFunction

    Defines a user-defined function of 5 arguments as user-defined function (UDF).

    Defines a user-defined function of 5 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  69. def udf[RT, A1, A2, A3, A4](f: (A1, A2, A3, A4) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3], arg4: scala.reflect.api.JavaUniverse.TypeTag[A4]): UserDefinedFunction

    Defines a user-defined function of 4 arguments as user-defined function (UDF).

    Defines a user-defined function of 4 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  70. def udf[RT, A1, A2, A3](f: (A1, A2, A3) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2], arg3: scala.reflect.api.JavaUniverse.TypeTag[A3]): UserDefinedFunction

    Defines a user-defined function of 3 arguments as user-defined function (UDF).

    Defines a user-defined function of 3 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  71. def udf[RT, A1, A2](f: (A1, A2) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1], arg2: scala.reflect.api.JavaUniverse.TypeTag[A2]): UserDefinedFunction

    Defines a user-defined function of 2 arguments as user-defined function (UDF).

    Defines a user-defined function of 2 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  72. def udf[RT, A1](f: (A1) ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT], arg1: scala.reflect.api.JavaUniverse.TypeTag[A1]): UserDefinedFunction

    Defines a user-defined function of 1 arguments as user-defined function (UDF).

    Defines a user-defined function of 1 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  73. def udf[RT](f: () ⇒ RT)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[RT]): UserDefinedFunction

    Defines a user-defined function of 0 arguments as user-defined function (UDF).

    Defines a user-defined function of 0 arguments as user-defined function (UDF). The data types are automatically inferred based on the function's signature.

  74. def upper(e: Column): Column

    Converts a string expression to upper case.

  75. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  76. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  77. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Aggregate functions

Non-aggregate functions

Sorting functions

UDF functions

Ungrouped