Packages

c

org.apache.spark.sql

DataFrameWriter

abstract class DataFrameWriter[T] extends AnyRef

Interface used to write a org.apache.spark.sql.api.Dataset to external storage systems (e.g. file systems, key-value stores, etc). Use Dataset.write to access this.

Annotations
@Stable()
Source
DataFrameWriter.scala
Since

1.4.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DataFrameWriter
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new DataFrameWriter()

Abstract Value Members

  1. abstract def insertInto(tableName: String): Unit

    Inserts the content of the DataFrame to the specified table.

    Inserts the content of the DataFrame to the specified table. It requires that the schema of the DataFrame is the same as the schema of the table.

    Since

    1.4.0

    Note

    Unlike saveAsTable, insertInto ignores the column names and just uses position-based resolution. For example:

    ,

    SaveMode.ErrorIfExists and SaveMode.Ignore behave as SaveMode.Append in insertInto as insertInto is not a table creating operation.

    scala> Seq((1, 2)).toDF("i", "j").write.mode("overwrite").saveAsTable("t1")
scala> Seq((3, 4)).toDF("j", "i").write.insertInto("t1")
scala> Seq((5, 6)).toDF("a", "b").write.insertInto("t1")
scala> sql("select * from t1").show
+---+---+
|  i|  j|
+---+---+
|  5|  6|
|  3|  4|
|  1|  2|
+---+---+

    Because it inserts data to an existing table, format or options will be ignored.

  2. abstract def save(): Unit

    Saves the content of the DataFrame as the specified table.

    Saves the content of the DataFrame as the specified table.

    Since

    1.4.0

  3. abstract def save(path: String): Unit

    Saves the content of the DataFrame at the specified path.

    Saves the content of the DataFrame at the specified path.

    Since

    1.4.0

  4. abstract def saveAsTable(tableName: String): Unit

    Saves the content of the DataFrame as the specified table.

    Saves the content of the DataFrame as the specified table.

    In the case the table already exists, behavior of this function depends on the save mode, specified by the mode function (default to throwing an exception). When mode is Overwrite, the schema of the DataFrame does not need to be the same as that of the existing table.

    When mode is Append, if there is an existing table, we will use the format and options of the existing table. The column order in the schema of the DataFrame doesn't need to be same as that of the existing table. Unlike insertInto, saveAsTable will use the column names to find the correct column positions. For example:

    scala> Seq((1, 2)).toDF("i", "j").write.mode("overwrite").saveAsTable("t1")
scala> Seq((3, 4)).toDF("j", "i").write.mode("append").saveAsTable("t1")
scala> sql("select * from t1").show
+---+---+
|  i|  j|
+---+---+
|  1|  2|
|  4|  3|
+---+---+

    In this method, save mode is used to determine the behavior if the data source table exists in Spark catalog. We will always overwrite the underlying data of data source (e.g. a table in JDBC data source) if the table doesn't exist in Spark catalog, and will always append to the underlying data of data source if the table already exists.

    When the DataFrame is created from a non-partitioned HadoopFsRelation with a single input path, and the data source provider can be mapped to an existing Hive builtin SerDe (i.e. ORC and Parquet), the table is persisted in a Hive compatible format, which means other systems like Hive will be able to read this table. Otherwise, the table is persisted in a Spark SQL specific format.

    Since

    1.4.0

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def assertNotBucketed(operation: String): Unit
    Attributes
    protected
  6. def assertNotClustered(operation: String): Unit
    Attributes
    protected
  7. def assertNotPartitioned(operation: String): Unit
    Attributes
    protected
  8. def bucketBy(numBuckets: Int, colName: String, colNames: String*): DataFrameWriter.this.type

    Buckets the output by the given columns.

    Buckets the output by the given columns. If specified, the output is laid out on the file system similar to Hive's bucketing scheme, but with a different bucket hash function and is not compatible with Hive's bucketing.

    This is applicable for all file-based data sources (e.g. Parquet, JSON) starting with Spark 2.1.0.

    Annotations
    @varargs()
    Since

    2.0

  9. var bucketColumnNames: Option[Seq[String]]
    Attributes
    protected
  10. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  11. def clusterBy(colName: String, colNames: String*): DataFrameWriter.this.type

    Clusters the output by the given columns on the storage.

    Clusters the output by the given columns on the storage. The rows with matching values in the specified clustering columns will be consolidated within the same group.

    For instance, if you cluster a dataset by date, the data sharing the same date will be stored together in a file. This arrangement improves query efficiency when you apply selective filters to these clustering columns, thanks to data skipping.

    Annotations
    @varargs()
    Since

    4.0

  12. var clusteringColumns: Option[Seq[String]]
    Attributes
    protected
  13. def csv(path: String): Unit

    Saves the content of the DataFrame in CSV format at the specified path.

    Saves the content of the DataFrame in CSV format at the specified path. This is equivalent to:

    format("csv").save(path)

    You can find the CSV-specific options for writing CSV files in <a href="https://spark.apache.org/docs/latest/sql-data-sources-csv.html#data-source-option"> Data Source Option in the version you use.

    Since

    2.0.0

  14. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  16. var extraOptions: CaseInsensitiveMap[String]
    Attributes
    protected
  17. def format(source: String): DataFrameWriter.this.type

    Specifies the underlying output data source.

    Specifies the underlying output data source. Built-in options include "parquet", "json", etc.

    Since

    1.4.0

  18. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  20. def isBucketed(): Boolean
    Attributes
    protected
  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. def jdbc(url: String, table: String, connectionProperties: Properties): Unit

    Saves the content of the DataFrame to an external database table via JDBC.

    Saves the content of the DataFrame to an external database table via JDBC. In the case the table already exists in the external database, behavior of this function depends on the save mode, specified by the mode function (default to throwing an exception).

    Don't create too many partitions in parallel on a large cluster; otherwise Spark might crash your external database systems.

    JDBC-specific option and parameter documentation for storing tables via JDBC in <a href="https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html#data-source-option"> Data Source Option in the version you use.

    table

    Name of the table in the external database.

    connectionProperties

    JDBC database connection arguments, a list of arbitrary string tag/value. Normally at least a "user" and "password" property should be included. "batchsize" can be used to control the number of rows per insert. "isolationLevel" can be one of "NONE", "READ_COMMITTED", "READ_UNCOMMITTED", "REPEATABLE_READ", or "SERIALIZABLE", corresponding to standard transaction isolation levels defined by JDBC's Connection object, with default of "READ_UNCOMMITTED".

    Since

    1.4.0

  23. def json(path: String): Unit

    Saves the content of the DataFrame in JSON format ( JSON Lines text format or newline-delimited JSON) at the specified path.

    Saves the content of the DataFrame in JSON format ( JSON Lines text format or newline-delimited JSON) at the specified path. This is equivalent to:

    format("json").save(path)

    You can find the JSON-specific options for writing JSON files in <a href="https://spark.apache.org/docs/latest/sql-data-sources-json.html#data-source-option"> Data Source Option in the version you use.

    Since

    1.4.0

  24. def mode(saveMode: String): DataFrameWriter.this.type

    Specifies the behavior when data or table already exists.

    Specifies the behavior when data or table already exists. Options include:

    • overwrite: overwrite the existing data.
    • append: append the data.
    • ignore: ignore the operation (i.e. no-op).
    • error or errorifexists: default option, throw an exception at runtime.
    Since

    1.4.0

  25. def mode(saveMode: SaveMode): DataFrameWriter.this.type

    Specifies the behavior when data or table already exists.

    Specifies the behavior when data or table already exists. Options include:

    • SaveMode.Overwrite: overwrite the existing data.
    • SaveMode.Append: append the data.
    • SaveMode.Ignore: ignore the operation (i.e. no-op).
    • SaveMode.ErrorIfExists: throw an exception at runtime.

    The default option is ErrorIfExists.

    Since

    1.4.0

  26. var mode: SaveMode
    Attributes
    protected
  27. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  28. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  29. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  30. var numBuckets: Option[Int]
    Attributes
    protected
  31. def option(key: String, value: Double): DataFrameWriter.this.type

    Adds an output option for the underlying data source.

    Adds an output option for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    2.0.0

  32. def option(key: String, value: Long): DataFrameWriter.this.type

    Adds an output option for the underlying data source.

    Adds an output option for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    2.0.0

  33. def option(key: String, value: Boolean): DataFrameWriter.this.type

    Adds an output option for the underlying data source.

    Adds an output option for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    2.0.0

  34. def option(key: String, value: String): DataFrameWriter.this.type

    Adds an output option for the underlying data source.

    Adds an output option for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    1.4.0

  35. def options(options: Map[String, String]): DataFrameWriter.this.type

    Adds output options for the underlying data source.

    Adds output options for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    1.4.0

  36. def options(options: Map[String, String]): DataFrameWriter.this.type

    (Scala-specific) Adds output options for the underlying data source.

    (Scala-specific) Adds output options for the underlying data source.

    All options are maintained in a case-insensitive way in terms of key names. If a new option has the same key case-insensitively, it will override the existing option.

    Since

    1.4.0

  37. def orc(path: String): Unit

    Saves the content of the DataFrame in ORC format at the specified path.

    Saves the content of the DataFrame in ORC format at the specified path. This is equivalent to:

    format("orc").save(path)

    ORC-specific option(s) for writing ORC files can be found in Data Source Option in the version you use.

    Since

    1.5.0

  38. def parquet(path: String): Unit

    Saves the content of the DataFrame in Parquet format at the specified path.

    Saves the content of the DataFrame in Parquet format at the specified path. This is equivalent to:

    format("parquet").save(path)

    Parquet-specific option(s) for writing Parquet files can be found in Data Source Option in the version you use.

    Since

    1.4.0

  39. def partitionBy(colNames: String*): DataFrameWriter.this.type

    Partitions the output by the given columns on the file system.

    Partitions the output by the given columns on the file system. If specified, the output is laid out on the file system similar to Hive's partitioning scheme. As an example, when we partition a dataset by year and then month, the directory layout would look like:

    • year=2016/month=01/
    • year=2016/month=02/

    Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.

    This is applicable for all file-based data sources (e.g. Parquet, JSON) starting with Spark 2.1.0.

    Annotations
    @varargs()
    Since

    1.4.0

  40. var partitioningColumns: Option[Seq[String]]
    Attributes
    protected
  41. def sortBy(colName: String, colNames: String*): DataFrameWriter.this.type

    Sorts the output in each bucket by the given columns.

    Sorts the output in each bucket by the given columns.

    This is applicable for all file-based data sources (e.g. Parquet, JSON) starting with Spark 2.1.0.

    Annotations
    @varargs()
    Since

    2.0

  42. var sortColumnNames: Option[Seq[String]]
    Attributes
    protected
  43. var source: String
    Attributes
    protected
  44. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  45. def text(path: String): Unit

    Saves the content of the DataFrame in a text file at the specified path.

    Saves the content of the DataFrame in a text file at the specified path. The DataFrame must have only one column that is of string type. Each row becomes a new line in the output file. For example:

    // Scala:
df.write.text("/path/to/output")

// Java:
df.write().text("/path/to/output")

    The text files will be encoded as UTF-8.

    You can find the text-specific options for writing text files in <a href="https://spark.apache.org/docs/latest/sql-data-sources-text.html#data-source-option"> Data Source Option in the version you use.

    Since

    1.6.0

  46. def toString(): String
    Definition Classes
    AnyRef → Any
  47. def validatePartitioning(): Unit

    Validate that clusterBy is not used with partitionBy or bucketBy.

    Validate that clusterBy is not used with partitionBy or bucketBy.

    Attributes
    protected
  48. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  49. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  50. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  51. def xml(path: String): Unit

    Saves the content of the DataFrame in XML format at the specified path.

    Saves the content of the DataFrame in XML format at the specified path. This is equivalent to:

    format("xml").save(path)

    Note that writing a XML file from DataFrame having a field ArrayType with its element as ArrayType would have an additional nested field for the element. For example, the DataFrame having a field below,

    fieldA [[data1], [data2]]

    would produce a XML file below. data1 data2

    Namely, roundtrip in writing and reading can end up in different schema structure.

    You can find the XML-specific options for writing XML files in <a href="https://spark.apache.org/docs/latest/sql-data-sources-xml.html#data-source-option"> Data Source Option in the version you use.

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AnyRef

Inherited from Any

Ungrouped