Interface CreateTableWriter<T>
- All Superinterfaces:
WriteConfigMethods<CreateTableWriter<T>>
- All Known Implementing Classes:
DataFrameWriterV2
- Since:
- 3.0.0
-
Method Summary
Modifier and TypeMethodDescriptionClusters the output by the given columns on the storage.Clusters the output by the given columns on the storage.void
create()
void
Create a new table or replace an existing table with the contents of the data frame.partitionedBy
(Column column, Column... columns) Partition the output table created bycreate
,createOrReplace
, orreplace
using the given columns or transforms.partitionedBy
(Column column, scala.collection.immutable.Seq<Column> columns) Partition the output table created bycreate
,createOrReplace
, orreplace
using the given columns or transforms.void
replace()
Replace an existing table with the contents of the data frame.tableProperty
(String property, String value) Add a table property.Specifies a provider for the underlying output data source.
-
Method Details
-
clusterBy
Clusters the output by the given columns on the storage. The rows with matching values in the specified clustering columns will be consolidated within the same group.For instance, if you cluster a dataset by date, the data sharing the same date will be stored together in a file. This arrangement improves query efficiency when you apply selective filters to these clustering columns, thanks to data skipping.
- Parameters:
colName
- (undocumented)colNames
- (undocumented)- Returns:
- (undocumented)
- Since:
- 4.0.0
-
clusterBy
Clusters the output by the given columns on the storage. The rows with matching values in the specified clustering columns will be consolidated within the same group.For instance, if you cluster a dataset by date, the data sharing the same date will be stored together in a file. This arrangement improves query efficiency when you apply selective filters to these clustering columns, thanks to data skipping.
- Parameters:
colName
- (undocumented)colNames
- (undocumented)- Returns:
- (undocumented)
- Since:
- 4.0.0
-
create
void create() throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException- Throws:
org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
-
createOrReplace
void createOrReplace()Create a new table or replace an existing table with the contents of the data frame.The output table's schema, partition layout, properties, and other configuration will be based on the contents of the data frame and the configuration set on this writer. If the table exists, its configuration and data will be replaced.
-
partitionedBy
Partition the output table created bycreate
,createOrReplace
, orreplace
using the given columns or transforms.When specified, the table data will be stored by these values for efficient reads.
For example, when a table is partitioned by day, it may be stored in a directory layout like:
table/day=2019-06-01/
table/day=2019-06-02/
Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.
- Parameters:
column
- (undocumented)columns
- (undocumented)- Returns:
- (undocumented)
- Since:
- 3.0.0
-
partitionedBy
Partition the output table created bycreate
,createOrReplace
, orreplace
using the given columns or transforms.When specified, the table data will be stored by these values for efficient reads.
For example, when a table is partitioned by day, it may be stored in a directory layout like:
table/day=2019-06-01/
table/day=2019-06-02/
Partitioning is one of the most widely used techniques to optimize physical data layout. It provides a coarse-grained index for skipping unnecessary data reads when queries have predicates on the partitioned columns. In order for partitioning to work well, the number of distinct values in each column should typically be less than tens of thousands.
- Parameters:
column
- (undocumented)columns
- (undocumented)- Returns:
- (undocumented)
- Since:
- 3.0.0
-
replace
void replace() throws org.apache.spark.sql.catalyst.analysis.CannotReplaceMissingTableExceptionReplace an existing table with the contents of the data frame.The existing table's schema, partition layout, properties, and other configuration will be replaced with the contents of the data frame and the configuration set on this writer.
If the output table does not exist, this operation will fail with
CannotReplaceMissingTableException
.- Throws:
org.apache.spark.sql.catalyst.analysis.CannotReplaceMissingTableException
- If the table does not exist
-
tableProperty
Add a table property.- Parameters:
property
- (undocumented)value
- (undocumented)- Returns:
- (undocumented)
-
using
Specifies a provider for the underlying output data source. Spark's default catalog supports "parquet", "json", etc.- Parameters:
provider
- (undocumented)- Returns:
- (undocumented)
- Since:
- 3.0.0
-