Class

org.apache.spark.sql

Dataset

Related Doc: package sql

Permalink

class Dataset[T] extends Serializable

A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations. Each Dataset also has an untyped view called a DataFrame, which is a Dataset of Row.

Operations available on Datasets are divided into transformations and actions. Transformations are the ones that produce new Datasets, and actions are the ones that trigger computation and return results. Example transformations include map, filter, select, and aggregate (groupBy). Example actions count, show, or writing data out to file systems.

Datasets are "lazy", i.e. computations are only triggered when an action is invoked. Internally, a Dataset represents a logical plan that describes the computation required to produce the data. When an action is invoked, Spark's query optimizer optimizes the logical plan and generates a physical plan for efficient execution in a parallel and distributed manner. To explore the logical plan as well as optimized physical plan, use the explain function.

To efficiently support domain-specific objects, an Encoder is required. The encoder maps the domain specific type T to Spark's internal type system. For example, given a class Person with two fields, name (string) and age (int), an encoder is used to tell Spark to generate code at runtime to serialize the Person object into a binary structure. This binary structure often has much lower memory footprint as well as are optimized for efficiency in data processing (e.g. in a columnar format). To understand the internal binary representation for data, use the schema function.

There are typically two ways to create a Dataset. The most common way is by pointing Spark to some files on storage systems, using the read function available on a SparkSession.

val people = spark.read.parquet("...").as[Person]  // Scala
Dataset<Person> people = spark.read().parquet("...").as(Encoders.bean(Person.class)); // Java

Datasets can also be created through transformations available on existing Datasets. For example, the following creates a new Dataset by applying a filter on the existing one:

val names = people.map(_.name)  // in Scala; names is a Dataset[String]
Dataset<String> names = people.map((Person p) -> p.name, Encoders.STRING));

Dataset operations can also be untyped, through various domain-specific-language (DSL) functions defined in: Dataset (this class), Column, and functions. These operations are very similar to the operations available in the data frame abstraction in R or Python.

To select a column from the Dataset, use apply method in Scala and col in Java.

val ageCol = people("age")  // in Scala
Column ageCol = people.col("age"); // in Java

Note that the Column type can also be manipulated through its various functions.

// The following creates a new column that increases everybody's age by 10.
people("age") + 10  // in Scala
people.col("age").plus(10);  // in Java

A more concrete example in Scala:

// To create Dataset[Row] using SparkSession
val people = spark.read.parquet("...")
val department = spark.read.parquet("...")

people.filter("age > 30")
  .join(department, people("deptId") === department("id"))
  .groupBy(department("name"), people("gender"))
  .agg(avg(people("salary")), max(people("age")))

and in Java:

// To create Dataset using SparkSession
Dataset<Row> people = spark.read().parquet("...");
Dataset<Row> department = spark.read().parquet("...");

people.filter(people.col("age").gt(30))
  .join(department, people.col("deptId").equalTo(department.col("id")))
  .groupBy(department.col("name"), people.col("gender"))
  .agg(avg(people.col("salary")), max(people.col("age")));
Annotations
@Stable()
Source
Dataset.scala
Since

1.6.0

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. Dataset
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Dataset(sqlContext: SQLContext, logicalPlan: LogicalPlan, encoder: Encoder[T])

    Permalink
  2. new Dataset(sparkSession: SparkSession, logicalPlan: LogicalPlan, encoder: Encoder[T])

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def agg(expr: Column, exprs: Column*): DataFrame

    Permalink

    Aggregates on the entire Dataset without groups.

    Aggregates on the entire Dataset without groups.

    // ds.agg(...) is a shorthand for ds.groupBy().agg(...)
    ds.agg(max($"age"), avg($"salary"))
    ds.groupBy().agg(max($"age"), avg($"salary"))
    Annotations
    @varargs()
    Since

    2.0.0

  5. def agg(exprs: Map[String, String]): DataFrame

    Permalink

    (Java-specific) Aggregates on the entire Dataset without groups.

    (Java-specific) Aggregates on the entire Dataset without groups.

    // ds.agg(...) is a shorthand for ds.groupBy().agg(...)
    ds.agg(Map("age" -> "max", "salary" -> "avg"))
    ds.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
    Since

    2.0.0

  6. def agg(exprs: Map[String, String]): DataFrame

    Permalink

    (Scala-specific) Aggregates on the entire Dataset without groups.

    (Scala-specific) Aggregates on the entire Dataset without groups.

    // ds.agg(...) is a shorthand for ds.groupBy().agg(...)
    ds.agg(Map("age" -> "max", "salary" -> "avg"))
    ds.groupBy().agg(Map("age" -> "max", "salary" -> "avg"))
    Since

    2.0.0

  7. def agg(aggExpr: (String, String), aggExprs: (String, String)*): DataFrame

    Permalink

    (Scala-specific) Aggregates on the entire Dataset without groups.

    (Scala-specific) Aggregates on the entire Dataset without groups.

    // ds.agg(...) is a shorthand for ds.groupBy().agg(...)
    ds.agg("age" -> "max", "salary" -> "avg")
    ds.groupBy().agg("age" -> "max", "salary" -> "avg")
    Since

    2.0.0

  8. def alias(alias: Symbol): Dataset[T]

    Permalink

    (Scala-specific) Returns a new Dataset with an alias set.

    (Scala-specific) Returns a new Dataset with an alias set. Same as as.

    Since

    2.0.0

  9. def alias(alias: String): Dataset[T]

    Permalink

    Returns a new Dataset with an alias set.

    Returns a new Dataset with an alias set. Same as as.

    Since

    2.0.0

  10. def apply(colName: String): Column

    Permalink

    Selects column based on the column name and returns it as a Column.

    Selects column based on the column name and returns it as a Column.

    Since

    2.0.0

    Note

    The column name can also reference to a nested column like a.b.

  11. def as(alias: Symbol): Dataset[T]

    Permalink

    (Scala-specific) Returns a new Dataset with an alias set.

    (Scala-specific) Returns a new Dataset with an alias set.

    Since

    2.0.0

  12. def as(alias: String): Dataset[T]

    Permalink

    Returns a new Dataset with an alias set.

    Returns a new Dataset with an alias set.

    Since

    1.6.0

  13. def as[U](implicit arg0: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: Returns a new Dataset where each record has been mapped on to the specified type.

    :: Experimental :: Returns a new Dataset where each record has been mapped on to the specified type. The method used to map columns depend on the type of U:

    • When U is a class, fields for the class will be mapped to columns of the same name (case sensitivity is determined by spark.sql.caseSensitive).
    • When U is a tuple, the columns will be mapped by ordinal (i.e. the first column will be assigned to _1).
    • When U is a primitive type (i.e. String, Int, etc), then the first column of the DataFrame will be used.

    If the schema of the Dataset does not match the desired U type, you can use select along with alias or as to rearrange or rename as required.

    Note that as[] only changes the view of the data that is passed into typed operations, such as map(), and does not eagerly project away any columns that are not present in the specified class.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  14. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  15. def cache(): Dataset.this.type

    Permalink

    Persist this Dataset with the default storage level (MEMORY_AND_DISK).

    Persist this Dataset with the default storage level (MEMORY_AND_DISK).

    Since

    1.6.0

  16. def checkpoint(eager: Boolean): Dataset[T]

    Permalink

    Returns a checkpointed version of this Dataset.

    Returns a checkpointed version of this Dataset. Checkpointing can be used to truncate the logical plan of this Dataset, which is especially useful in iterative algorithms where the plan may grow exponentially. It will be saved to files inside the checkpoint directory set with SparkContext#setCheckpointDir.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  17. def checkpoint(): Dataset[T]

    Permalink

    Eagerly checkpoint a Dataset and return the new Dataset.

    Eagerly checkpoint a Dataset and return the new Dataset. Checkpointing can be used to truncate the logical plan of this Dataset, which is especially useful in iterative algorithms where the plan may grow exponentially. It will be saved to files inside the checkpoint directory set with SparkContext#setCheckpointDir.

    Annotations
    @Experimental() @Evolving()
    Since

    2.1.0

  18. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  19. def coalesce(numPartitions: Int): Dataset[T]

    Permalink

    Returns a new Dataset that has exactly numPartitions partitions, when the fewer partitions are requested.

    Returns a new Dataset that has exactly numPartitions partitions, when the fewer partitions are requested. If a larger number of partitions is requested, it will stay at the current number of partitions. Similar to coalesce defined on an RDD, this operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions.

    However, if you're doing a drastic coalesce, e.g. to numPartitions = 1, this may result in your computation taking place on fewer nodes than you like (e.g. one node in the case of numPartitions = 1). To avoid this, you can call repartition. This will add a shuffle step, but means the current upstream partitions will be executed in parallel (per whatever the current partitioning is).

    Since

    1.6.0

  20. def col(colName: String): Column

    Permalink

    Selects column based on the column name and returns it as a Column.

    Selects column based on the column name and returns it as a Column.

    Since

    2.0.0

    Note

    The column name can also reference to a nested column like a.b.

  21. def colRegex(colName: String): Column

    Permalink

    Selects column based on the column name specified as a regex and returns it as Column.

    Selects column based on the column name specified as a regex and returns it as Column.

    Since

    2.3.0

  22. def collect(): Array[T]

    Permalink

    Returns an array that contains all rows in this Dataset.

    Returns an array that contains all rows in this Dataset.

    Running collect requires moving all the data into the application's driver process, and doing so on a very large dataset can crash the driver process with OutOfMemoryError.

    For Java API, use collectAsList.

    Since

    1.6.0

  23. def collectAsList(): List[T]

    Permalink

    Returns a Java list that contains all rows in this Dataset.

    Returns a Java list that contains all rows in this Dataset.

    Running collect requires moving all the data into the application's driver process, and doing so on a very large dataset can crash the driver process with OutOfMemoryError.

    Since

    1.6.0

  24. def columns: Array[String]

    Permalink

    Returns all column names as an array.

    Returns all column names as an array.

    Since

    1.6.0

  25. def count(): Long

    Permalink

    Returns the number of rows in the Dataset.

    Returns the number of rows in the Dataset.

    Since

    1.6.0

  26. def createGlobalTempView(viewName: String): Unit

    Permalink

    Creates a global temporary view using the given name.

    Creates a global temporary view using the given name. The lifetime of this temporary view is tied to this Spark application.

    Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

    Annotations
    @throws( ... )
    Since

    2.1.0

    Exceptions thrown

    AnalysisException if the view name is invalid or already exists

  27. def createOrReplaceGlobalTempView(viewName: String): Unit

    Permalink

    Creates or replaces a global temporary view using the given name.

    Creates or replaces a global temporary view using the given name. The lifetime of this temporary view is tied to this Spark application.

    Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

    Since

    2.2.0

  28. def createOrReplaceTempView(viewName: String): Unit

    Permalink

    Creates a local temporary view using the given name.

    Creates a local temporary view using the given name. The lifetime of this temporary view is tied to the SparkSession that was used to create this Dataset.

    Since

    2.0.0

  29. def createTempView(viewName: String): Unit

    Permalink

    Creates a local temporary view using the given name.

    Creates a local temporary view using the given name. The lifetime of this temporary view is tied to the SparkSession that was used to create this Dataset.

    Local temporary view is session-scoped. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

    Annotations
    @throws( ... )
    Since

    2.0.0

    Exceptions thrown

    AnalysisException if the view name is invalid or already exists

  30. def crossJoin(right: Dataset[_]): DataFrame

    Permalink

    Explicit cartesian join with another DataFrame.

    Explicit cartesian join with another DataFrame.

    right

    Right side of the join operation.

    Since

    2.1.0

    Note

    Cartesian joins are very expensive without an extra filter that can be pushed down.

  31. def cube(col1: String, cols: String*): RelationalGroupedDataset

    Permalink

    Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.

    Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    This is a variant of cube that can only group by existing columns using column names (i.e. cannot construct expressions).

    // Compute the average for all numeric columns cubed by department and group.
    ds.cube("department", "group").avg()
    
    // Compute the max age and average salary, cubed by department and gender.
    ds.cube($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  32. def cube(cols: Column*): RelationalGroupedDataset

    Permalink

    Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.

    Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    // Compute the average for all numeric columns cubed by department and group.
    ds.cube($"department", $"group").avg()
    
    // Compute the max age and average salary, cubed by department and gender.
    ds.cube($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  33. def describe(cols: String*): DataFrame

    Permalink

    Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max.

    Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max. If no columns are given, this function computes statistics for all numerical or string columns.

    This function is meant for exploratory data analysis, as we make no guarantee about the backward compatibility of the schema of the resulting Dataset. If you want to programmatically compute summary statistics, use the agg function instead.

    ds.describe("age", "height").show()
    
    // output:
    // summary age   height
    // count   10.0  10.0
    // mean    53.3  178.05
    // stddev  11.6  15.7
    // min     18.0  163.0
    // max     92.0  192.0

    Use summary for expanded statistics and control over which statistics to compute.

    cols

    Columns to compute statistics on.

    Annotations
    @varargs()
    Since

    1.6.0

  34. def distinct(): Dataset[T]

    Permalink

    Returns a new Dataset that contains only the unique rows from this Dataset.

    Returns a new Dataset that contains only the unique rows from this Dataset. This is an alias for dropDuplicates.

    Since

    2.0.0

    Note

    Equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

  35. def drop(col: Column): DataFrame

    Permalink

    Returns a new Dataset with a column dropped.

    Returns a new Dataset with a column dropped. This version of drop accepts a Column rather than a name. This is a no-op if the Dataset doesn't have a column with an equivalent expression.

    Since

    2.0.0

  36. def drop(colNames: String*): DataFrame

    Permalink

    Returns a new Dataset with columns dropped.

    Returns a new Dataset with columns dropped. This is a no-op if schema doesn't contain column name(s).

    This method can only be used to drop top level columns. the colName string is treated literally without further interpretation.

    Annotations
    @varargs()
    Since

    2.0.0

  37. def drop(colName: String): DataFrame

    Permalink

    Returns a new Dataset with a column dropped.

    Returns a new Dataset with a column dropped. This is a no-op if schema doesn't contain column name.

    This method can only be used to drop top level columns. the colName string is treated literally without further interpretation.

    Since

    2.0.0

  38. def dropDuplicates(col1: String, cols: String*): Dataset[T]

    Permalink

    Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    For a static batch Dataset, it just drops duplicate rows. For a streaming Dataset, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark to limit how late the duplicate data can be and system will accordingly limit the state. In addition, too late data older than watermark will be dropped to avoid any possibility of duplicates.

    Annotations
    @varargs()
    Since

    2.0.0

  39. def dropDuplicates(colNames: Array[String]): Dataset[T]

    Permalink

    Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    For a static batch Dataset, it just drops duplicate rows. For a streaming Dataset, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark to limit how late the duplicate data can be and system will accordingly limit the state. In addition, too late data older than watermark will be dropped to avoid any possibility of duplicates.

    Since

    2.0.0

  40. def dropDuplicates(colNames: Seq[String]): Dataset[T]

    Permalink

    (Scala-specific) Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    (Scala-specific) Returns a new Dataset with duplicate rows removed, considering only the subset of columns.

    For a static batch Dataset, it just drops duplicate rows. For a streaming Dataset, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark to limit how late the duplicate data can be and system will accordingly limit the state. In addition, too late data older than watermark will be dropped to avoid any possibility of duplicates.

    Since

    2.0.0

  41. def dropDuplicates(): Dataset[T]

    Permalink

    Returns a new Dataset that contains only the unique rows from this Dataset.

    Returns a new Dataset that contains only the unique rows from this Dataset. This is an alias for distinct.

    For a static batch Dataset, it just drops duplicate rows. For a streaming Dataset, it will keep all data across triggers as intermediate state to drop duplicates rows. You can use withWatermark to limit how late the duplicate data can be and system will accordingly limit the state. In addition, too late data older than watermark will be dropped to avoid any possibility of duplicates.

    Since

    2.0.0

  42. def dtypes: Array[(String, String)]

    Permalink

    Returns all column names and their data types as an array.

    Returns all column names and their data types as an array.

    Since

    1.6.0

  43. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  44. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  45. def except(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing rows in this Dataset but not in another Dataset.

    Returns a new Dataset containing rows in this Dataset but not in another Dataset. This is equivalent to EXCEPT DISTINCT in SQL.

    Since

    2.0.0

    Note

    Equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

  46. def exceptAll(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing rows in this Dataset but not in another Dataset while preserving the duplicates.

    Returns a new Dataset containing rows in this Dataset but not in another Dataset while preserving the duplicates. This is equivalent to EXCEPT ALL in SQL.

    Since

    2.4.0

    Note

    Equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T. Also as standard in SQL, this function resolves columns by position (not by name).

  47. def explain(): Unit

    Permalink

    Prints the physical plan to the console for debugging purposes.

    Prints the physical plan to the console for debugging purposes.

    Since

    1.6.0

  48. def explain(extended: Boolean): Unit

    Permalink

    Prints the plans (logical and physical) to the console for debugging purposes.

    Prints the plans (logical and physical) to the console for debugging purposes.

    Since

    1.6.0

  49. def filter(func: FilterFunction[T]): Dataset[T]

    Permalink

    :: Experimental :: (Java-specific) Returns a new Dataset that only contains elements where func returns true.

    :: Experimental :: (Java-specific) Returns a new Dataset that only contains elements where func returns true.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  50. def filter(func: (T) ⇒ Boolean): Dataset[T]

    Permalink

    :: Experimental :: (Scala-specific) Returns a new Dataset that only contains elements where func returns true.

    :: Experimental :: (Scala-specific) Returns a new Dataset that only contains elements where func returns true.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  51. def filter(conditionExpr: String): Dataset[T]

    Permalink

    Filters rows using the given SQL expression.

    Filters rows using the given SQL expression.

    peopleDs.filter("age > 15")
    Since

    1.6.0

  52. def filter(condition: Column): Dataset[T]

    Permalink

    Filters rows using the given condition.

    Filters rows using the given condition.

    // The following are equivalent:
    peopleDs.filter($"age" > 15)
    peopleDs.where($"age" > 15)
    Since

    1.6.0

  53. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  54. def first(): T

    Permalink

    Returns the first row.

    Returns the first row. Alias for head().

    Since

    1.6.0

  55. def flatMap[U](f: FlatMapFunction[T, U], encoder: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Java-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.

    :: Experimental :: (Java-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  56. def flatMap[U](func: (T) ⇒ TraversableOnce[U])(implicit arg0: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Scala-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.

    :: Experimental :: (Scala-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  57. def foreach(func: ForeachFunction[T]): Unit

    Permalink

    (Java-specific) Runs func on each element of this Dataset.

    (Java-specific) Runs func on each element of this Dataset.

    Since

    1.6.0

  58. def foreach(f: (T) ⇒ Unit): Unit

    Permalink

    Applies a function f to all rows.

    Applies a function f to all rows.

    Since

    1.6.0

  59. def foreachPartition(func: ForeachPartitionFunction[T]): Unit

    Permalink

    (Java-specific) Runs func on each partition of this Dataset.

    (Java-specific) Runs func on each partition of this Dataset.

    Since

    1.6.0

  60. def foreachPartition(f: (Iterator[T]) ⇒ Unit): Unit

    Permalink

    Applies a function f to each partition of this Dataset.

    Applies a function f to each partition of this Dataset.

    Since

    1.6.0

  61. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  62. def groupBy(col1: String, cols: String*): RelationalGroupedDataset

    Permalink

    Groups the Dataset using the specified columns, so that we can run aggregation on them.

    Groups the Dataset using the specified columns, so that we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    This is a variant of groupBy that can only group by existing columns using column names (i.e. cannot construct expressions).

    // Compute the average for all numeric columns grouped by department.
    ds.groupBy("department").avg()
    
    // Compute the max age and average salary, grouped by department and gender.
    ds.groupBy($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  63. def groupBy(cols: Column*): RelationalGroupedDataset

    Permalink

    Groups the Dataset using the specified columns, so we can run aggregation on them.

    Groups the Dataset using the specified columns, so we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    // Compute the average for all numeric columns grouped by department.
    ds.groupBy($"department").avg()
    
    // Compute the max age and average salary, grouped by department and gender.
    ds.groupBy($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  64. def groupByKey[K](func: MapFunction[T, K], encoder: Encoder[K]): KeyValueGroupedDataset[K, T]

    Permalink

    :: Experimental :: (Java-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.

    :: Experimental :: (Java-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  65. def groupByKey[K](func: (T) ⇒ K)(implicit arg0: Encoder[K]): KeyValueGroupedDataset[K, T]

    Permalink

    :: Experimental :: (Scala-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.

    :: Experimental :: (Scala-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.

    Annotations
    @Experimental() @Evolving()
    Since

    2.0.0

  66. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  67. def head(): T

    Permalink

    Returns the first row.

    Returns the first row.

    Since

    1.6.0

  68. def head(n: Int): Array[T]

    Permalink

    Returns the first n rows.

    Returns the first n rows.

    Since

    1.6.0

    Note

    this method should only be used if the resulting array is expected to be small, as all the data is loaded into the driver's memory.

  69. def hint(name: String, parameters: Any*): Dataset[T]

    Permalink

    Specifies some hint on the current Dataset.

    Specifies some hint on the current Dataset. As an example, the following code specifies that one of the plan can be broadcasted:

    df1.join(df2.hint("broadcast"))
    Annotations
    @varargs()
    Since

    2.2.0

  70. def inputFiles: Array[String]

    Permalink

    Returns a best-effort snapshot of the files that compose this Dataset.

    Returns a best-effort snapshot of the files that compose this Dataset. This method simply asks each constituent BaseRelation for its respective files and takes the union of all results. Depending on the source relations, this may not find all input files. Duplicates are removed.

    Since

    2.0.0

  71. def intersect(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing rows only in both this Dataset and another Dataset.

    Returns a new Dataset containing rows only in both this Dataset and another Dataset. This is equivalent to INTERSECT in SQL.

    Since

    1.6.0

    Note

    Equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T.

  72. def intersectAll(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing rows only in both this Dataset and another Dataset while preserving the duplicates.

    Returns a new Dataset containing rows only in both this Dataset and another Dataset while preserving the duplicates. This is equivalent to INTERSECT ALL in SQL.

    Since

    2.4.0

    Note

    Equality checking is performed directly on the encoded representation of the data and thus is not affected by a custom equals function defined on T. Also as standard in SQL, this function resolves columns by position (not by name).

  73. def isEmpty: Boolean

    Permalink

    Returns true if the Dataset is empty.

    Returns true if the Dataset is empty.

    Since

    2.4.0

  74. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  75. def isLocal: Boolean

    Permalink

    Returns true if the collect and take methods can be run locally (without any Spark executors).

    Returns true if the collect and take methods can be run locally (without any Spark executors).

    Since

    1.6.0

  76. def isStreaming: Boolean

    Permalink

    Returns true if this Dataset contains one or more sources that continuously return data as it arrives.

    Returns true if this Dataset contains one or more sources that continuously return data as it arrives. A Dataset that reads data from a streaming source must be executed as a StreamingQuery using the start() method in DataStreamWriter. Methods that return a single answer, e.g. count() or collect(), will throw an AnalysisException when there is a streaming source present.

    Annotations
    @Evolving()
    Since

    2.0.0

  77. def javaRDD: JavaRDD[T]

    Permalink

    Returns the content of the Dataset as a JavaRDD of Ts.

    Returns the content of the Dataset as a JavaRDD of Ts.

    Since

    1.6.0

  78. def join(right: Dataset[_], joinExprs: Column, joinType: String): DataFrame

    Permalink

    Join with another DataFrame, using the given join expression.

    Join with another DataFrame, using the given join expression. The following performs a full outer join between df1 and df2.

    // Scala:
    import org.apache.spark.sql.functions._
    df1.join(df2, $"df1Key" === $"df2Key", "outer")
    
    // Java:
    import static org.apache.spark.sql.functions.*;
    df1.join(df2, col("df1Key").equalTo(col("df2Key")), "outer");
    right

    Right side of the join.

    joinExprs

    Join expression.

    joinType

    Type of join to perform. Default inner. Must be one of: inner, cross, outer, full, full_outer, left, left_outer, right, right_outer, left_semi, left_anti.

    Since

    2.0.0

  79. def join(right: Dataset[_], joinExprs: Column): DataFrame

    Permalink

    Inner join with another DataFrame, using the given join expression.

    Inner join with another DataFrame, using the given join expression.

    // The following two are equivalent:
    df1.join(df2, $"df1Key" === $"df2Key")
    df1.join(df2).where($"df1Key" === $"df2Key")
    Since

    2.0.0

  80. def join(right: Dataset[_], usingColumns: Seq[String], joinType: String): DataFrame

    Permalink

    Equi-join with another DataFrame using the given columns.

    Equi-join with another DataFrame using the given columns. A cross join with a predicate is specified as an inner join. If you would explicitly like to perform a cross join use the crossJoin method.

    Different from other join functions, the join columns will only appear once in the output, i.e. similar to SQL's JOIN USING syntax.

    right

    Right side of the join operation.

    usingColumns

    Names of the columns to join on. This columns must exist on both sides.

    joinType

    Type of join to perform. Default inner. Must be one of: inner, cross, outer, full, full_outer, left, left_outer, right, right_outer, left_semi, left_anti.

    Since

    2.0.0

    Note

    If you perform a self-join using this function without aliasing the input DataFrames, you will NOT be able to reference any columns after the join, since there is no way to disambiguate which side of the join you would like to reference.

  81. def join(right: Dataset[_], usingColumns: Seq[String]): DataFrame

    Permalink

    Inner equi-join with another DataFrame using the given columns.

    Inner equi-join with another DataFrame using the given columns.

    Different from other join functions, the join columns will only appear once in the output, i.e. similar to SQL's JOIN USING syntax.

    // Joining df1 and df2 using the columns "user_id" and "user_name"
    df1.join(df2, Seq("user_id", "user_name"))
    right

    Right side of the join operation.

    usingColumns

    Names of the columns to join on. This columns must exist on both sides.

    Since

    2.0.0

    Note

    If you perform a self-join using this function without aliasing the input DataFrames, you will NOT be able to reference any columns after the join, since there is no way to disambiguate which side of the join you would like to reference.

  82. def join(right: Dataset[_], usingColumn: String): DataFrame

    Permalink

    Inner equi-join with another DataFrame using the given column.

    Inner equi-join with another DataFrame using the given column.

    Different from other join functions, the join column will only appear once in the output, i.e. similar to SQL's JOIN USING syntax.

    // Joining df1 and df2 using the column "user_id"
    df1.join(df2, "user_id")
    right

    Right side of the join operation.

    usingColumn

    Name of the column to join on. This column must exist on both sides.

    Since

    2.0.0

    Note

    If you perform a self-join using this function without aliasing the input DataFrames, you will NOT be able to reference any columns after the join, since there is no way to disambiguate which side of the join you would like to reference.

  83. def join(right: Dataset[_]): DataFrame

    Permalink

    Join with another DataFrame.

    Join with another DataFrame.

    Behaves as an INNER JOIN and requires a subsequent join predicate.

    right

    Right side of the join operation.

    Since

    2.0.0

  84. def joinWith[U](other: Dataset[U], condition: Column): Dataset[(T, U)]

    Permalink

    :: Experimental :: Using inner equi-join to join this Dataset returning a Tuple2 for each pair where condition evaluates to true.

    :: Experimental :: Using inner equi-join to join this Dataset returning a Tuple2 for each pair where condition evaluates to true.

    other

    Right side of the join.

    condition

    Join expression.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  85. def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)]

    Permalink

    :: Experimental :: Joins this Dataset returning a Tuple2 for each pair where condition evaluates to true.

    :: Experimental :: Joins this Dataset returning a Tuple2 for each pair where condition evaluates to true.

    This is similar to the relation join function with one important difference in the result schema. Since joinWith preserves objects present on either side of the join, the result schema is similarly nested into a tuple under the column names _1 and _2.

    This type of join can be useful both for preserving type-safety with the original object types as well as working with relational data where either side of the join has column names in common.

    other

    Right side of the join.

    condition

    Join expression.

    joinType

    Type of join to perform. Default inner. Must be one of: inner, cross, outer, full, full_outer, left, left_outer, right, right_outer.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  86. def limit(n: Int): Dataset[T]

    Permalink

    Returns a new Dataset by taking the first n rows.

    Returns a new Dataset by taking the first n rows. The difference between this function and head is that head is an action and returns an array (by triggering query execution) while limit returns a new Dataset.

    Since

    2.0.0

  87. def localCheckpoint(eager: Boolean): Dataset[T]

    Permalink

    Locally checkpoints a Dataset and return the new Dataset.

    Locally checkpoints a Dataset and return the new Dataset. Checkpointing can be used to truncate the logical plan of this Dataset, which is especially useful in iterative algorithms where the plan may grow exponentially. Local checkpoints are written to executor storage and despite potentially faster they are unreliable and may compromise job completion.

    Annotations
    @Experimental() @Evolving()
    Since

    2.3.0

  88. def localCheckpoint(): Dataset[T]

    Permalink

    Eagerly locally checkpoints a Dataset and return the new Dataset.

    Eagerly locally checkpoints a Dataset and return the new Dataset. Checkpointing can be used to truncate the logical plan of this Dataset, which is especially useful in iterative algorithms where the plan may grow exponentially. Local checkpoints are written to executor storage and despite potentially faster they are unreliable and may compromise job completion.

    Annotations
    @Experimental() @Evolving()
    Since

    2.3.0

  89. def map[U](func: MapFunction[T, U], encoder: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Java-specific) Returns a new Dataset that contains the result of applying func to each element.

    :: Experimental :: (Java-specific) Returns a new Dataset that contains the result of applying func to each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  90. def map[U](func: (T) ⇒ U)(implicit arg0: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Scala-specific) Returns a new Dataset that contains the result of applying func to each element.

    :: Experimental :: (Scala-specific) Returns a new Dataset that contains the result of applying func to each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  91. def mapPartitions[U](f: MapPartitionsFunction[T, U], encoder: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Java-specific) Returns a new Dataset that contains the result of applying f to each partition.

    :: Experimental :: (Java-specific) Returns a new Dataset that contains the result of applying f to each partition.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  92. def mapPartitions[U](func: (Iterator[T]) ⇒ Iterator[U])(implicit arg0: Encoder[U]): Dataset[U]

    Permalink

    :: Experimental :: (Scala-specific) Returns a new Dataset that contains the result of applying func to each partition.

    :: Experimental :: (Scala-specific) Returns a new Dataset that contains the result of applying func to each partition.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  93. def na: DataFrameNaFunctions

    Permalink

    Returns a DataFrameNaFunctions for working with missing data.

    Returns a DataFrameNaFunctions for working with missing data.

    // Dropping rows containing any null values.
    ds.na.drop()
    Since

    1.6.0

  94. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  95. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  96. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  97. def orderBy(sortExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset sorted by the given expressions.

    Returns a new Dataset sorted by the given expressions. This is an alias of the sort function.

    Annotations
    @varargs()
    Since

    2.0.0

  98. def orderBy(sortCol: String, sortCols: String*): Dataset[T]

    Permalink

    Returns a new Dataset sorted by the given expressions.

    Returns a new Dataset sorted by the given expressions. This is an alias of the sort function.

    Annotations
    @varargs()
    Since

    2.0.0

  99. def persist(newLevel: StorageLevel): Dataset.this.type

    Permalink

    Persist this Dataset with the given storage level.

    Persist this Dataset with the given storage level.

    newLevel

    One of: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, MEMORY_AND_DISK_SER, DISK_ONLY, MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc.

    Since

    1.6.0

  100. def persist(): Dataset.this.type

    Permalink

    Persist this Dataset with the default storage level (MEMORY_AND_DISK).

    Persist this Dataset with the default storage level (MEMORY_AND_DISK).

    Since

    1.6.0

  101. def printSchema(): Unit

    Permalink

    Prints the schema to the console in a nice tree format.

    Prints the schema to the console in a nice tree format.

    Since

    1.6.0

  102. val queryExecution: QueryExecution

    Permalink
  103. def randomSplit(weights: Array[Double]): Array[Dataset[T]]

    Permalink

    Randomly splits this Dataset with the provided weights.

    Randomly splits this Dataset with the provided weights.

    weights

    weights for splits, will be normalized if they don't sum to 1.

    Since

    2.0.0

  104. def randomSplit(weights: Array[Double], seed: Long): Array[Dataset[T]]

    Permalink

    Randomly splits this Dataset with the provided weights.

    Randomly splits this Dataset with the provided weights.

    weights

    weights for splits, will be normalized if they don't sum to 1.

    seed

    Seed for sampling. For Java API, use randomSplitAsList.

    Since

    2.0.0

  105. def randomSplitAsList(weights: Array[Double], seed: Long): List[Dataset[T]]

    Permalink

    Returns a Java list that contains randomly split Dataset with the provided weights.

    Returns a Java list that contains randomly split Dataset with the provided weights.

    weights

    weights for splits, will be normalized if they don't sum to 1.

    seed

    Seed for sampling.

    Since

    2.0.0

  106. lazy val rdd: RDD[T]

    Permalink

    Represents the content of the Dataset as an RDD of T.

    Represents the content of the Dataset as an RDD of T.

    Since

    1.6.0

  107. def reduce(func: ReduceFunction[T]): T

    Permalink

    :: Experimental :: (Java-specific) Reduces the elements of this Dataset using the specified binary function.

    :: Experimental :: (Java-specific) Reduces the elements of this Dataset using the specified binary function. The given func must be commutative and associative or the result may be non-deterministic.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  108. def reduce(func: (T, T) ⇒ T): T

    Permalink

    :: Experimental :: (Scala-specific) Reduces the elements of this Dataset using the specified binary function.

    :: Experimental :: (Scala-specific) Reduces the elements of this Dataset using the specified binary function. The given func must be commutative and associative or the result may be non-deterministic.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  109. def repartition(partitionExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.

    Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions. The resulting Dataset is hash partitioned.

    This is the same operation as "DISTRIBUTE BY" in SQL (Hive QL).

    Annotations
    @varargs()
    Since

    2.0.0

  110. def repartition(numPartitions: Int, partitionExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.

    Returns a new Dataset partitioned by the given partitioning expressions into numPartitions. The resulting Dataset is hash partitioned.

    This is the same operation as "DISTRIBUTE BY" in SQL (Hive QL).

    Annotations
    @varargs()
    Since

    2.0.0

  111. def repartition(numPartitions: Int): Dataset[T]

    Permalink

    Returns a new Dataset that has exactly numPartitions partitions.

    Returns a new Dataset that has exactly numPartitions partitions.

    Since

    1.6.0

  112. def repartitionByRange(partitionExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.

    Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions. The resulting Dataset is range partitioned.

    At least one partition-by expression must be specified. When no explicit sort order is specified, "ascending nulls first" is assumed. Note, the rows are not sorted in each partition of the resulting Dataset.

    Annotations
    @varargs()
    Since

    2.3.0

  113. def repartitionByRange(numPartitions: Int, partitionExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.

    Returns a new Dataset partitioned by the given partitioning expressions into numPartitions. The resulting Dataset is range partitioned.

    At least one partition-by expression must be specified. When no explicit sort order is specified, "ascending nulls first" is assumed. Note, the rows are not sorted in each partition of the resulting Dataset.

    Annotations
    @varargs()
    Since

    2.3.0

  114. def rollup(col1: String, cols: String*): RelationalGroupedDataset

    Permalink

    Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.

    Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    This is a variant of rollup that can only group by existing columns using column names (i.e. cannot construct expressions).

    // Compute the average for all numeric columns rolluped by department and group.
    ds.rollup("department", "group").avg()
    
    // Compute the max age and average salary, rolluped by department and gender.
    ds.rollup($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  115. def rollup(cols: Column*): RelationalGroupedDataset

    Permalink

    Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.

    Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them. See RelationalGroupedDataset for all the available aggregate functions.

    // Compute the average for all numeric columns rolluped by department and group.
    ds.rollup($"department", $"group").avg()
    
    // Compute the max age and average salary, rolluped by department and gender.
    ds.rollup($"department", $"gender").agg(Map(
      "salary" -> "avg",
      "age" -> "max"
    ))
    Annotations
    @varargs()
    Since

    2.0.0

  116. def sample(withReplacement: Boolean, fraction: Double): Dataset[T]

    Permalink

    Returns a new Dataset by sampling a fraction of rows, using a random seed.

    Returns a new Dataset by sampling a fraction of rows, using a random seed.

    withReplacement

    Sample with replacement or not.

    fraction

    Fraction of rows to generate, range [0.0, 1.0].

    Since

    1.6.0

    Note

    This is NOT guaranteed to provide exactly the fraction of the total count of the given Dataset.

  117. def sample(withReplacement: Boolean, fraction: Double, seed: Long): Dataset[T]

    Permalink

    Returns a new Dataset by sampling a fraction of rows, using a user-supplied seed.

    Returns a new Dataset by sampling a fraction of rows, using a user-supplied seed.

    withReplacement

    Sample with replacement or not.

    fraction

    Fraction of rows to generate, range [0.0, 1.0].

    seed

    Seed for sampling.

    Since

    1.6.0

    Note

    This is NOT guaranteed to provide exactly the fraction of the count of the given Dataset.

  118. def sample(fraction: Double): Dataset[T]

    Permalink

    Returns a new Dataset by sampling a fraction of rows (without replacement), using a random seed.

    Returns a new Dataset by sampling a fraction of rows (without replacement), using a random seed.

    fraction

    Fraction of rows to generate, range [0.0, 1.0].

    Since

    2.3.0

    Note

    This is NOT guaranteed to provide exactly the fraction of the count of the given Dataset.

  119. def sample(fraction: Double, seed: Long): Dataset[T]

    Permalink

    Returns a new Dataset by sampling a fraction of rows (without replacement), using a user-supplied seed.

    Returns a new Dataset by sampling a fraction of rows (without replacement), using a user-supplied seed.

    fraction

    Fraction of rows to generate, range [0.0, 1.0].

    seed

    Seed for sampling.

    Since

    2.3.0

    Note

    This is NOT guaranteed to provide exactly the fraction of the count of the given Dataset.

  120. def schema: StructType

    Permalink

    Returns the schema of this Dataset.

    Returns the schema of this Dataset.

    Since

    1.6.0

  121. def select[U1, U2, U3, U4, U5](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3], c4: TypedColumn[T, U4], c5: TypedColumn[T, U5]): Dataset[(U1, U2, U3, U4, U5)]

    Permalink

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  122. def select[U1, U2, U3, U4](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3], c4: TypedColumn[T, U4]): Dataset[(U1, U2, U3, U4)]

    Permalink

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  123. def select[U1, U2, U3](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2], c3: TypedColumn[T, U3]): Dataset[(U1, U2, U3)]

    Permalink

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  124. def select[U1, U2](c1: TypedColumn[T, U1], c2: TypedColumn[T, U2]): Dataset[(U1, U2)]

    Permalink

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    :: Experimental :: Returns a new Dataset by computing the given Column expressions for each element.

    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  125. def select[U1](c1: TypedColumn[T, U1]): Dataset[U1]

    Permalink

    :: Experimental :: Returns a new Dataset by computing the given Column expression for each element.

    :: Experimental :: Returns a new Dataset by computing the given Column expression for each element.

    val ds = Seq(1, 2, 3).toDS()
    val newDS = ds.select(expr("value + 1").as[Int])
    Annotations
    @Experimental() @Evolving()
    Since

    1.6.0

  126. def select(col: String, cols: String*): DataFrame

    Permalink

    Selects a set of columns.

    Selects a set of columns. This is a variant of select that can only select existing columns using column names (i.e. cannot construct expressions).

    // The following two are equivalent:
    ds.select("colA", "colB")
    ds.select($"colA", $"colB")
    Annotations
    @varargs()
    Since

    2.0.0

  127. def select(cols: Column*): DataFrame

    Permalink

    Selects a set of column based expressions.

    Selects a set of column based expressions.

    ds.select($"colA", $"colB" + 1)
    Annotations
    @varargs()
    Since

    2.0.0

  128. def selectExpr(exprs: String*): DataFrame

    Permalink

    Selects a set of SQL expressions.

    Selects a set of SQL expressions. This is a variant of select that accepts SQL expressions.

    // The following are equivalent:
    ds.selectExpr("colA", "colB as newName", "abs(colC)")
    ds.select(expr("colA"), expr("colB as newName"), expr("abs(colC)"))
    Annotations
    @varargs()
    Since

    2.0.0

  129. def selectUntyped(columns: TypedColumn[_, _]*): Dataset[_]

    Permalink

    Internal helper function for building typed selects that return tuples.

    Internal helper function for building typed selects that return tuples. For simplicity and code reuse, we do this without the help of the type system and then use helper functions that cast appropriately for the user facing interface.

    Attributes
    protected
  130. def show(numRows: Int, truncate: Int, vertical: Boolean): Unit

    Permalink

    Displays the Dataset in a tabular form.

    Displays the Dataset in a tabular form. For example:

    year  month AVG('Adj Close) MAX('Adj Close)
    1980  12    0.503218        0.595103
    1981  01    0.523289        0.570307
    1982  02    0.436504        0.475256
    1983  03    0.410516        0.442194
    1984  04    0.450090        0.483521

    If vertical enabled, this command prints output rows vertically (one line per column value)?

    -RECORD 0-------------------
     year            | 1980
     month           | 12
     AVG('Adj Close) | 0.503218
     AVG('Adj Close) | 0.595103
    -RECORD 1-------------------
     year            | 1981
     month           | 01
     AVG('Adj Close) | 0.523289
     AVG('Adj Close) | 0.570307
    -RECORD 2-------------------
     year            | 1982
     month           | 02
     AVG('Adj Close) | 0.436504
     AVG('Adj Close) | 0.475256
    -RECORD 3-------------------
     year            | 1983
     month           | 03
     AVG('Adj Close) | 0.410516
     AVG('Adj Close) | 0.442194
    -RECORD 4-------------------
     year            | 1984
     month           | 04
     AVG('Adj Close) | 0.450090
     AVG('Adj Close) | 0.483521
    numRows

    Number of rows to show

    truncate

    If set to more than 0, truncates strings to truncate characters and all cells will be aligned right.

    vertical

    If set to true, prints output rows vertically (one line per column value).

    Since

    2.3.0

  131. def show(numRows: Int, truncate: Int): Unit

    Permalink

    Displays the Dataset in a tabular form.

    Displays the Dataset in a tabular form. For example:

    year  month AVG('Adj Close) MAX('Adj Close)
    1980  12    0.503218        0.595103
    1981  01    0.523289        0.570307
    1982  02    0.436504        0.475256
    1983  03    0.410516        0.442194
    1984  04    0.450090        0.483521
    numRows

    Number of rows to show

    truncate

    If set to more than 0, truncates strings to truncate characters and all cells will be aligned right.

    Since

    1.6.0

  132. def show(numRows: Int, truncate: Boolean): Unit

    Permalink

    Displays the Dataset in a tabular form.

    Displays the Dataset in a tabular form. For example:

    year  month AVG('Adj Close) MAX('Adj Close)
    1980  12    0.503218        0.595103
    1981  01    0.523289        0.570307
    1982  02    0.436504        0.475256
    1983  03    0.410516        0.442194
    1984  04    0.450090        0.483521
    numRows

    Number of rows to show

    truncate

    Whether truncate long strings. If true, strings more than 20 characters will be truncated and all cells will be aligned right

    Since

    1.6.0

  133. def show(truncate: Boolean): Unit

    Permalink

    Displays the top 20 rows of Dataset in a tabular form.

    Displays the top 20 rows of Dataset in a tabular form.

    truncate

    Whether truncate long strings. If true, strings more than 20 characters will be truncated and all cells will be aligned right

    Since

    1.6.0

  134. def show(): Unit

    Permalink

    Displays the top 20 rows of Dataset in a tabular form.

    Displays the top 20 rows of Dataset in a tabular form. Strings more than 20 characters will be truncated, and all cells will be aligned right.

    Since

    1.6.0

  135. def show(numRows: Int): Unit

    Permalink

    Displays the Dataset in a tabular form.

    Displays the Dataset in a tabular form. Strings more than 20 characters will be truncated, and all cells will be aligned right. For example:

    year  month AVG('Adj Close) MAX('Adj Close)
    1980  12    0.503218        0.595103
    1981  01    0.523289        0.570307
    1982  02    0.436504        0.475256
    1983  03    0.410516        0.442194
    1984  04    0.450090        0.483521
    numRows

    Number of rows to show

    Since

    1.6.0

  136. def sort(sortExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset sorted by the given expressions.

    Returns a new Dataset sorted by the given expressions. For example:

    ds.sort($"col1", $"col2".desc)
    Annotations
    @varargs()
    Since

    2.0.0

  137. def sort(sortCol: String, sortCols: String*): Dataset[T]

    Permalink

    Returns a new Dataset sorted by the specified column, all in ascending order.

    Returns a new Dataset sorted by the specified column, all in ascending order.

    // The following 3 are equivalent
    ds.sort("sortcol")
    ds.sort($"sortcol")
    ds.sort($"sortcol".asc)
    Annotations
    @varargs()
    Since

    2.0.0

  138. def sortWithinPartitions(sortExprs: Column*): Dataset[T]

    Permalink

    Returns a new Dataset with each partition sorted by the given expressions.

    Returns a new Dataset with each partition sorted by the given expressions.

    This is the same operation as "SORT BY" in SQL (Hive QL).

    Annotations
    @varargs()
    Since

    2.0.0

  139. def sortWithinPartitions(sortCol: String, sortCols: String*): Dataset[T]

    Permalink

    Returns a new Dataset with each partition sorted by the given expressions.

    Returns a new Dataset with each partition sorted by the given expressions.

    This is the same operation as "SORT BY" in SQL (Hive QL).

    Annotations
    @varargs()
    Since

    2.0.0

  140. val sparkSession: SparkSession

    Permalink
  141. lazy val sqlContext: SQLContext

    Permalink
  142. def stat: DataFrameStatFunctions

    Permalink

    Returns a DataFrameStatFunctions for working statistic functions support.

    Returns a DataFrameStatFunctions for working statistic functions support.

    // Finding frequent items in column with name 'a'.
    ds.stat.freqItems(Seq("a"))
    Since

    1.6.0

  143. def storageLevel: StorageLevel

    Permalink

    Get the Dataset's current storage level, or StorageLevel.NONE if not persisted.

    Get the Dataset's current storage level, or StorageLevel.NONE if not persisted.

    Since

    2.1.0

  144. def summary(statistics: String*): DataFrame

    Permalink

    Computes specified statistics for numeric and string columns.

    Computes specified statistics for numeric and string columns. Available statistics are:

    - count - mean - stddev - min - max - arbitrary approximate percentiles specified as a percentage (eg, 75%)

    If no statistics are given, this function computes count, mean, stddev, min, approximate quartiles (percentiles at 25%, 50%, and 75%), and max.

    This function is meant for exploratory data analysis, as we make no guarantee about the backward compatibility of the schema of the resulting Dataset. If you want to programmatically compute summary statistics, use the agg function instead.

    ds.summary().show()
    
    // output:
    // summary age   height
    // count   10.0  10.0
    // mean    53.3  178.05
    // stddev  11.6  15.7
    // min     18.0  163.0
    // 25%     24.0  176.0
    // 50%     24.0  176.0
    // 75%     32.0  180.0
    // max     92.0  192.0
    ds.summary("count", "min", "25%", "75%", "max").show()
    
    // output:
    // summary age   height
    // count   10.0  10.0
    // min     18.0  163.0
    // 25%     24.0  176.0
    // 75%     32.0  180.0
    // max     92.0  192.0

    To do a summary for specific columns first select them:

    ds.select("age", "height").summary().show()

    See also describe for basic statistics.

    statistics

    Statistics from above list to be computed.

    Annotations
    @varargs()
    Since

    2.3.0

  145. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  146. def take(n: Int): Array[T]

    Permalink

    Returns the first n rows in the Dataset.

    Returns the first n rows in the Dataset.

    Running take requires moving data into the application's driver process, and doing so with a very large n can crash the driver process with OutOfMemoryError.

    Since

    1.6.0

  147. def takeAsList(n: Int): List[T]

    Permalink

    Returns the first n rows in the Dataset as a list.

    Returns the first n rows in the Dataset as a list.

    Running take requires moving data into the application's driver process, and doing so with a very large n can crash the driver process with OutOfMemoryError.

    Since

    1.6.0

  148. def toDF(colNames: String*): DataFrame

    Permalink

    Converts this strongly typed collection of data to generic DataFrame with columns renamed.

    Converts this strongly typed collection of data to generic DataFrame with columns renamed. This can be quite convenient in conversion from an RDD of tuples into a DataFrame with meaningful names. For example:

    val rdd: RDD[(Int, String)] = ...
    rdd.toDF()  // this implicit conversion creates a DataFrame with column name `_1` and `_2`
    rdd.toDF("id", "name")  // this creates a DataFrame with column name "id" and "name"
    Annotations
    @varargs()
    Since

    2.0.0

  149. def toDF(): DataFrame

    Permalink

    Converts this strongly typed collection of data to generic Dataframe.

    Converts this strongly typed collection of data to generic Dataframe. In contrast to the strongly typed objects that Dataset operations work on, a Dataframe returns generic Row objects that allow fields to be accessed by ordinal or name.

    Since

    1.6.0

  150. def toJSON: Dataset[String]

    Permalink

    Returns the content of the Dataset as a Dataset of JSON strings.

    Returns the content of the Dataset as a Dataset of JSON strings.

    Since

    2.0.0

  151. def toJavaRDD: JavaRDD[T]

    Permalink

    Returns the content of the Dataset as a JavaRDD of Ts.

    Returns the content of the Dataset as a JavaRDD of Ts.

    Since

    1.6.0

  152. def toLocalIterator(): Iterator[T]

    Permalink

    Returns an iterator that contains all rows in this Dataset.

    Returns an iterator that contains all rows in this Dataset.

    The iterator will consume as much memory as the largest partition in this Dataset.

    Since

    2.0.0

    Note

    this results in multiple Spark jobs, and if the input Dataset is the result of a wide transformation (e.g. join with different partitioners), to avoid recomputing the input Dataset should be cached first.

  153. def toString(): String

    Permalink
    Definition Classes
    Dataset → AnyRef → Any
  154. def transform[U](t: (Dataset[T]) ⇒ Dataset[U]): Dataset[U]

    Permalink

    Concise syntax for chaining custom transformations.

    Concise syntax for chaining custom transformations.

    def featurize(ds: Dataset[T]): Dataset[U] = ...
    
    ds
      .transform(featurize)
      .transform(...)
    Since

    1.6.0

  155. def union(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    This is equivalent to UNION ALL in SQL. To do a SQL-style set union (that does deduplication of elements), use this function followed by a distinct.

    Also as standard in SQL, this function resolves columns by position (not by name):

    val df1 = Seq((1, 2, 3)).toDF("col0", "col1", "col2")
    val df2 = Seq((4, 5, 6)).toDF("col1", "col2", "col0")
    df1.union(df2).show
    
    // output:
    // +----+----+----+
    // |col0|col1|col2|
    // +----+----+----+
    // |   1|   2|   3|
    // |   4|   5|   6|
    // +----+----+----+

    Notice that the column positions in the schema aren't necessarily matched with the fields in the strongly typed objects in a Dataset. This function resolves columns by their positions in the schema, not the fields in the strongly typed objects. Use unionByName to resolve columns by field name in the typed objects.

    Since

    2.0.0

  156. def unionByName(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    This is different from both UNION ALL and UNION DISTINCT in SQL. To do a SQL-style set union (that does deduplication of elements), use this function followed by a distinct.

    The difference between this function and union is that this function resolves columns by name (not by position):

    val df1 = Seq((1, 2, 3)).toDF("col0", "col1", "col2")
    val df2 = Seq((4, 5, 6)).toDF("col1", "col2", "col0")
    df1.unionByName(df2).show
    
    // output:
    // +----+----+----+
    // |col0|col1|col2|
    // +----+----+----+
    // |   1|   2|   3|
    // |   6|   4|   5|
    // +----+----+----+
    Since

    2.3.0

  157. def unpersist(): Dataset.this.type

    Permalink

    Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk.

    Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk. This will not un-persist any cached data that is built upon this Dataset.

    Since

    1.6.0

  158. def unpersist(blocking: Boolean): Dataset.this.type

    Permalink

    Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk.

    Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk. This will not un-persist any cached data that is built upon this Dataset.

    blocking

    Whether to block until all blocks are deleted.

    Since

    1.6.0

  159. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  160. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  161. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  162. def where(conditionExpr: String): Dataset[T]

    Permalink

    Filters rows using the given SQL expression.

    Filters rows using the given SQL expression.

    peopleDs.where("age > 15")
    Since

    1.6.0

  163. def where(condition: Column): Dataset[T]

    Permalink

    Filters rows using the given condition.

    Filters rows using the given condition. This is an alias for filter.

    // The following are equivalent:
    peopleDs.filter($"age" > 15)
    peopleDs.where($"age" > 15)
    Since

    1.6.0

  164. def withColumn(colName: String, col: Column): DataFrame

    Permalink

    Returns a new Dataset by adding a column or replacing the existing column that has the same name.

    Returns a new Dataset by adding a column or replacing the existing column that has the same name.

    column's expression must only refer to attributes supplied by this Dataset. It is an error to add a column that refers to some other Dataset.

    Since

    2.0.0

  165. def withColumnRenamed(existingName: String, newName: String): DataFrame

    Permalink

    Returns a new Dataset with a column renamed.

    Returns a new Dataset with a column renamed. This is a no-op if schema doesn't contain existingName.

    Since

    2.0.0

  166. def withWatermark(eventTime: String, delayThreshold: String): Dataset[T]

    Permalink

    Defines an event time watermark for this Dataset.

    Defines an event time watermark for this Dataset. A watermark tracks a point in time before which we assume no more late data is going to arrive.

    Spark will use this watermark for several purposes:

    • To know when a given time window aggregation can be finalized and thus can be emitted when using output modes that do not allow updates.
    • To minimize the amount of state that we need to keep for on-going aggregations, mapGroupsWithState and dropDuplicates operators.

    The current watermark is computed by looking at the MAX(eventTime) seen across all of the partitions in the query minus a user specified delayThreshold. Due to the cost of coordinating this value across partitions, the actual watermark used is only guaranteed to be at least delayThreshold behind the actual event time. In some cases we may still process records that arrive more than delayThreshold late.

    eventTime

    the name of the column that contains the event time of the row.

    delayThreshold

    the minimum delay to wait to data to arrive late, relative to the latest record that has been processed in the form of an interval (e.g. "1 minute" or "5 hours"). NOTE: This should not be negative.

    Annotations
    @Evolving()
    Since

    2.1.0

  167. def write: DataFrameWriter[T]

    Permalink

    Interface for saving the content of the non-streaming Dataset out into external storage.

    Interface for saving the content of the non-streaming Dataset out into external storage.

    Since

    1.6.0

  168. def writeStream: DataStreamWriter[T]

    Permalink

    Interface for saving the content of the streaming Dataset out into external storage.

    Interface for saving the content of the streaming Dataset out into external storage.

    Annotations
    @Evolving()
    Since

    2.0.0

Deprecated Value Members

  1. def explode[A, B](inputColumn: String, outputColumn: String)(f: (A) ⇒ TraversableOnce[B])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[B]): DataFrame

    Permalink

    (Scala-specific) Returns a new Dataset where a single column has been expanded to zero or more rows by the provided function.

    (Scala-specific) Returns a new Dataset where a single column has been expanded to zero or more rows by the provided function. This is similar to a LATERAL VIEW in HiveQL. All columns of the input row are implicitly joined with each value that is output by the function.

    Given that this is deprecated, as an alternative, you can explode columns either using functions.explode():

    ds.select(explode(split('words, " ")).as("word"))

    or flatMap():

    ds.flatMap(_.words.split(" "))
    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) use flatMap() or select() with functions.explode() instead

    Since

    2.0.0

  2. def explode[A <: Product](input: Column*)(f: (Row) ⇒ TraversableOnce[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    Permalink

    (Scala-specific) Returns a new Dataset where each row has been expanded to zero or more rows by the provided function.

    (Scala-specific) Returns a new Dataset where each row has been expanded to zero or more rows by the provided function. This is similar to a LATERAL VIEW in HiveQL. The columns of the input row are implicitly joined with each row that is output by the function.

    Given that this is deprecated, as an alternative, you can explode columns either using functions.explode() or flatMap(). The following example uses these alternatives to count the number of books that contain a given word:

    case class Book(title: String, words: String)
    val ds: Dataset[Book]
    
    val allWords = ds.select('title, explode(split('words, " ")).as("word"))
    
    val bookCountPerWord = allWords.groupBy("word").agg(countDistinct("title"))

    Using flatMap() this can similarly be exploded as:

    ds.flatMap(_.words.split(" "))
    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) use flatMap() or select() with functions.explode() instead

    Since

    2.0.0

  3. def registerTempTable(tableName: String): Unit

    Permalink

    Registers this Dataset as a temporary table using the given name.

    Registers this Dataset as a temporary table using the given name. The lifetime of this temporary table is tied to the SparkSession that was used to create this Dataset.

    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) Use createOrReplaceTempView(viewName) instead.

    Since

    1.6.0

  4. def unionAll(other: Dataset[T]): Dataset[T]

    Permalink

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    Returns a new Dataset containing union of rows in this Dataset and another Dataset.

    This is equivalent to UNION ALL in SQL. To do a SQL-style set union (that does deduplication of elements), use this function followed by a distinct.

    Also as standard in SQL, this function resolves columns by position (not by name).

    Annotations
    @deprecated
    Deprecated

    (Since version 2.0.0) use union()

    Since

    2.0.0

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Actions

Basic Dataset functions

streaming

Typed transformations

Untyped transformations

Ungrouped