org.apache.spark.sql

GroupedData

class GroupedData extends AnyRef

:: Experimental :: A set of methods for aggregations on a DataFrame, created by DataFrame.groupBy.

Annotations
@Experimental()
Since

1.3.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. GroupedData
  2. AnyRef
  3. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GroupedData(df: DataFrame, groupingExprs: Seq[Expression], groupType: GroupType)

    Attributes
    protected[org.apache.spark.sql]

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def agg(expr: Column, exprs: Column*): DataFrame

    Compute aggregates by specifying a series of aggregate columns.

    Compute aggregates by specifying a series of aggregate columns. Note that this function by default retains the grouping columns in its output. To not retain grouping columns, set spark.sql.retainGroupColumns to false.

    The available aggregate methods are defined in org.apache.spark.sql.functions.

    // Selects the age of the oldest employee and the aggregate expense for each department
    
    // Scala:
    import org.apache.spark.sql.functions._
    df.groupBy("department").agg(max("age"), sum("expense"))
    
    // Java:
    import static org.apache.spark.sql.functions.*;
    df.groupBy("department").agg(max("age"), sum("expense"));

    Note that before Spark 1.4, the default behavior is to NOT retain grouping columns. To change to that behavior, set config variable spark.sql.retainGroupColumns to false.

    // Scala, 1.3.x:
    df.groupBy("department").agg($"department", max("age"), sum("expense"))
    
    // Java, 1.3.x:
    df.groupBy("department").agg(col("department"), max("age"), sum("expense"));
    Annotations
    @varargs()
    Since

    1.3.0

  7. def agg(exprs: Map[String, String]): DataFrame

    (Java-specific) Compute aggregates by specifying a map from column name to aggregate methods.

    (Java-specific) Compute aggregates by specifying a map from column name to aggregate methods. The resulting DataFrame will also contain the grouping columns.

    The available aggregate methods are avg, max, min, sum, count.

    // Selects the age of the oldest employee and the aggregate expense for each department
    import com.google.common.collect.ImmutableMap;
    df.groupBy("department").agg(ImmutableMap.of("age", "max", "expense", "sum"));
    Since

    1.3.0

  8. def agg(exprs: Map[String, String]): DataFrame

    (Scala-specific) Compute aggregates by specifying a map from column name to aggregate methods.

    (Scala-specific) Compute aggregates by specifying a map from column name to aggregate methods. The resulting DataFrame will also contain the grouping columns.

    The available aggregate methods are avg, max, min, sum, count.

    // Selects the age of the oldest employee and the aggregate expense for each department
    df.groupBy("department").agg(Map(
      "age" -> "max",
      "expense" -> "sum"
    ))
    Since

    1.3.0

  9. def agg(aggExpr: (String, String), aggExprs: (String, String)*): DataFrame

    (Scala-specific) Compute aggregates by specifying a map from column name to aggregate methods.

    (Scala-specific) Compute aggregates by specifying a map from column name to aggregate methods. The resulting DataFrame will also contain the grouping columns.

    The available aggregate methods are avg, max, min, sum, count.

    // Selects the age of the oldest employee and the aggregate expense for each department
    df.groupBy("department").agg(
      "age" -> "max",
      "expense" -> "sum"
    )
    Since

    1.3.0

  10. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  11. def avg(colNames: String*): DataFrame

    Compute the mean value for each numeric columns for each group.

    Compute the mean value for each numeric columns for each group. The resulting DataFrame will also contain the grouping columns. When specified columns are given, only compute the mean values for them.

    Annotations
    @varargs()
    Since

    1.3.0

  12. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  13. def count(): DataFrame

    Count the number of rows for each group.

    Count the number of rows for each group. The resulting DataFrame will also contain the grouping columns.

    Since

    1.3.0

  14. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  15. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  16. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  17. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  18. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  19. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  20. def max(colNames: String*): DataFrame

    Compute the max value for each numeric columns for each group.

    Compute the max value for each numeric columns for each group. The resulting DataFrame will also contain the grouping columns. When specified columns are given, only compute the max values for them.

    Annotations
    @varargs()
    Since

    1.3.0

  21. def mean(colNames: String*): DataFrame

    Compute the average value for each numeric columns for each group.

    Compute the average value for each numeric columns for each group. This is an alias for avg. The resulting DataFrame will also contain the grouping columns. When specified columns are given, only compute the average values for them.

    Annotations
    @varargs()
    Since

    1.3.0

  22. def min(colNames: String*): DataFrame

    Compute the min value for each numeric column for each group.

    Compute the min value for each numeric column for each group. The resulting DataFrame will also contain the grouping columns. When specified columns are given, only compute the min values for them.

    Annotations
    @varargs()
    Since

    1.3.0

  23. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. final def notify(): Unit

    Definition Classes
    AnyRef
  25. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  26. def sum(colNames: String*): DataFrame

    Compute the sum for each numeric columns for each group.

    Compute the sum for each numeric columns for each group. The resulting DataFrame will also contain the grouping columns. When specified columns are given, only compute the sum for them.

    Annotations
    @varargs()
    Since

    1.3.0

  27. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  28. def toString(): String

    Definition Classes
    AnyRef → Any
  29. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  30. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  31. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped