org.apache.spark.sql

Interface CreateTableWriter<T>

All Superinterfaces:

WriteConfigMethods<CreateTableWriter<T>>

All Known Implementing Classes:

DataFrameWriterV2

public interface CreateTableWriter<T>
extends WriteConfigMethods<CreateTableWriter<T>>

Trait to restrict calls to create and replace operations.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
void	create() Create a new table from the contents of the data frame.
void	createOrReplace() Create a new table or replace an existing table with the contents of the data frame.
CreateTableWriter<T>	partitionedBy(Column column, scala.collection.Seq<Column> columns) Partition the output table created by create, createOrReplace, or replace using the given columns or transforms.
void	replace() Replace an existing table with the contents of the data frame.
CreateTableWriter<T>	tableProperty(String property, String value) Add a table property.
CreateTableWriter<T>	using(String provider) Specifies a provider for the underlying output data source.

Methods inherited from interface org.apache.spark.sql.WriteConfigMethods

option, option, option, option, options, options

Method Detail

create

void create()
     throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException

Create a new table from the contents of the data frame.

The new table's schema, partition layout, properties, and other configuration will be based on the configuration set on this writer.

If the output table exists, this operation will fail with TableAlreadyExistsException.

Throws:

org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException - If the table already exists

createOrReplace

void createOrReplace()

Create a new table or replace an existing table with the contents of the data frame.

The output table's schema, partition layout, properties, and other configuration will be based on the contents of the data frame and the configuration set on this writer. If the table exists, its configuration and data will be replaced.

partitionedBy

CreateTableWriter<T> partitionedBy(Column column,
                                   scala.collection.Seq<Column> columns)

Partition the output table created by create, createOrReplace, or replace using the given columns or transforms.

When specified, the table data will be stored by these values for efficient reads.

For example, when a table is partitioned by day, it may be stored in a directory layout like:

• table/day=2019-06-01/
• table/day=2019-06-02/

Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.

Parameters:

column - (undocumented)

columns - (undocumented)

Returns:

(undocumented)

Since:

3.0.0

replace

void replace()
      throws org.apache.spark.sql.catalyst.analysis.CannotReplaceMissingTableException

Replace an existing table with the contents of the data frame.

The existing table's schema, partition layout, properties, and other configuration will be replaced with the contents of the data frame and the configuration set on this writer.

If the output table does not exist, this operation will fail with CannotReplaceMissingTableException.

Throws:

org.apache.spark.sql.catalyst.analysis.CannotReplaceMissingTableException - If the table does not exist

tableProperty

CreateTableWriter<T> tableProperty(String property,
                                   String value)

Add a table property.

Parameters:

property - (undocumented)

value - (undocumented)

Returns:

(undocumented)

using

CreateTableWriter<T> using(String provider)

Specifies a provider for the underlying output data source. Spark's default catalog supports "parquet", "json", etc.

Parameters:

provider - (undocumented)

Returns:

(undocumented)

Since:

3.0.0