public abstract class Catalog
extends Object
SparkSession.catalog
.
Constructor and Description |
---|
Catalog() |
Modifier and Type | Method and Description |
---|---|
abstract void |
cacheTable(String tableName)
Caches the specified table in-memory.
|
abstract void |
cacheTable(String tableName,
StorageLevel storageLevel)
Caches the specified table with the given storage level.
|
abstract void |
clearCache()
Removes all cached tables from the in-memory cache.
|
Dataset<Row> |
createExternalTable(String tableName,
String path)
Deprecated.
use createTable instead. Since 2.2.0.
|
Dataset<Row> |
createExternalTable(String tableName,
String source,
java.util.Map<String,String> options)
Deprecated.
use createTable instead. Since 2.2.0.
|
Dataset<Row> |
createExternalTable(String tableName,
String source,
scala.collection.immutable.Map<String,String> options)
Deprecated.
use createTable instead. Since 2.2.0.
|
Dataset<Row> |
createExternalTable(String tableName,
String path,
String source)
Deprecated.
use createTable instead. Since 2.2.0.
|
Dataset<Row> |
createExternalTable(String tableName,
String source,
StructType schema,
java.util.Map<String,String> options)
Deprecated.
use createTable instead. Since 2.2.0.
|
Dataset<Row> |
createExternalTable(String tableName,
String source,
StructType schema,
scala.collection.immutable.Map<String,String> options)
Deprecated.
use createTable instead. Since 2.2.0.
|
abstract Dataset<Row> |
createTable(String tableName,
String path)
Creates a table from the given path and returns the corresponding DataFrame.
|
Dataset<Row> |
createTable(String tableName,
String source,
java.util.Map<String,String> options)
Creates a table based on the dataset in a data source and a set of options.
|
abstract Dataset<Row> |
createTable(String tableName,
String source,
scala.collection.immutable.Map<String,String> options)
(Scala-specific)
Creates a table based on the dataset in a data source and a set of options.
|
abstract Dataset<Row> |
createTable(String tableName,
String path,
String source)
Creates a table from the given path based on a data source and returns the corresponding
DataFrame.
|
Dataset<Row> |
createTable(String tableName,
String source,
String description,
java.util.Map<String,String> options)
Creates a table based on the dataset in a data source and a set of options.
|
abstract Dataset<Row> |
createTable(String tableName,
String source,
String description,
scala.collection.immutable.Map<String,String> options)
(Scala-specific)
Creates a table based on the dataset in a data source and a set of options.
|
Dataset<Row> |
createTable(String tableName,
String source,
StructType schema,
java.util.Map<String,String> options)
Create a table based on the dataset in a data source, a schema and a set of options.
|
abstract Dataset<Row> |
createTable(String tableName,
String source,
StructType schema,
scala.collection.immutable.Map<String,String> options)
(Scala-specific)
Create a table based on the dataset in a data source, a schema and a set of options.
|
Dataset<Row> |
createTable(String tableName,
String source,
StructType schema,
String description,
java.util.Map<String,String> options)
Create a table based on the dataset in a data source, a schema and a set of options.
|
abstract Dataset<Row> |
createTable(String tableName,
String source,
StructType schema,
String description,
scala.collection.immutable.Map<String,String> options)
(Scala-specific)
Create a table based on the dataset in a data source, a schema and a set of options.
|
abstract String |
currentCatalog()
Returns the current catalog in this session.
|
abstract String |
currentDatabase()
Returns the current database (namespace) in this session.
|
abstract boolean |
databaseExists(String dbName)
Check if the database (namespace) with the specified name exists (the name can be qualified
with catalog).
|
abstract boolean |
dropGlobalTempView(String viewName)
Drops the global temporary view with the given view name in the catalog.
|
abstract boolean |
dropTempView(String viewName)
Drops the local temporary view with the given view name in the catalog.
|
abstract boolean |
functionExists(String functionName)
Check if the function with the specified name exists.
|
abstract boolean |
functionExists(String dbName,
String functionName)
Check if the function with the specified name exists in the specified database under the
Hive Metastore.
|
abstract Database |
getDatabase(String dbName)
Get the database (namespace) with the specified name (can be qualified with catalog).
|
abstract Function |
getFunction(String functionName)
Get the function with the specified name.
|
abstract Function |
getFunction(String dbName,
String functionName)
Get the function with the specified name in the specified database under the Hive Metastore.
|
abstract Table |
getTable(String tableName)
Get the table or view with the specified name.
|
abstract Table |
getTable(String dbName,
String tableName)
Get the table or view with the specified name in the specified database under the Hive
Metastore.
|
abstract boolean |
isCached(String tableName)
Returns true if the table is currently cached in-memory.
|
abstract Dataset<CatalogMetadata> |
listCatalogs()
Returns a list of catalogs available in this session.
|
abstract Dataset<CatalogMetadata> |
listCatalogs(String pattern)
Returns a list of catalogs which name match the specify pattern and available in this session.
|
abstract Dataset<Column> |
listColumns(String tableName)
Returns a list of columns for the given table/view or temporary view.
|
abstract Dataset<Column> |
listColumns(String dbName,
String tableName)
Returns a list of columns for the given table/view in the specified database under the Hive
Metastore.
|
abstract Dataset<Database> |
listDatabases()
Returns a list of databases (namespaces) available within the current catalog.
|
abstract Dataset<Database> |
listDatabases(String pattern)
Returns a list of databases (namespaces) which name match the specify pattern and
available within the current catalog.
|
abstract Dataset<Function> |
listFunctions()
Returns a list of functions registered in the current database (namespace).
|
abstract Dataset<Function> |
listFunctions(String dbName)
Returns a list of functions registered in the specified database (namespace) (the name can be
qualified with catalog).
|
abstract Dataset<Function> |
listFunctions(String dbName,
String pattern)
Returns a list of functions registered in the specified database (namespace)
which name match the specify pattern (the name can be qualified with catalog).
|
abstract Dataset<Table> |
listTables()
Returns a list of tables/views in the current database (namespace).
|
abstract Dataset<Table> |
listTables(String dbName)
Returns a list of tables/views in the specified database (namespace) (the name can be qualified
with catalog).
|
abstract Dataset<Table> |
listTables(String dbName,
String pattern)
Returns a list of tables/views in the specified database (namespace)
which name match the specify pattern (the name can be qualified with catalog).
|
abstract void |
recoverPartitions(String tableName)
Recovers all the partitions in the directory of a table and update the catalog.
|
abstract void |
refreshByPath(String path)
Invalidates and refreshes all the cached data (and the associated metadata) for any
Dataset
that contains the given data source path. |
abstract void |
refreshTable(String tableName)
Invalidates and refreshes all the cached data and metadata of the given table.
|
abstract void |
setCurrentCatalog(String catalogName)
Sets the current catalog in this session.
|
abstract void |
setCurrentDatabase(String dbName)
Sets the current database (namespace) in this session.
|
abstract boolean |
tableExists(String tableName)
Check if the table or view with the specified name exists.
|
abstract boolean |
tableExists(String dbName,
String tableName)
Check if the table or view with the specified name exists in the specified database under the
Hive Metastore.
|
abstract void |
uncacheTable(String tableName)
Removes the specified table from the in-memory cache.
|
public abstract void cacheTable(String tableName)
tableName
- is either a qualified or unqualified name that designates a table/view.
If no database identifier is provided, it refers to a temporary view or
a table/view in the current database.public abstract void cacheTable(String tableName, StorageLevel storageLevel)
tableName
- is either a qualified or unqualified name that designates a table/view.
If no database identifier is provided, it refers to a temporary view or
a table/view in the current database.storageLevel
- storage level to cache table.public abstract void clearCache()
public Dataset<Row> createExternalTable(String tableName, String path)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.path
- (undocumented)public Dataset<Row> createExternalTable(String tableName, String path, String source)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.path
- (undocumented)source
- (undocumented)public Dataset<Row> createExternalTable(String tableName, String source, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)options
- (undocumented)public Dataset<Row> createExternalTable(String tableName, String source, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)options
- (undocumented)public Dataset<Row> createExternalTable(String tableName, String source, StructType schema, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)options
- (undocumented)public Dataset<Row> createExternalTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)options
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String path)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.path
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String path, String source)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.path
- (undocumented)source
- (undocumented)public Dataset<Row> createTable(String tableName, String source, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)options
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String source, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)options
- (undocumented)public Dataset<Row> createTable(String tableName, String source, String description, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)description
- (undocumented)options
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String source, String description, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)description
- (undocumented)options
- (undocumented)public Dataset<Row> createTable(String tableName, String source, StructType schema, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)options
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String source, StructType schema, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)options
- (undocumented)public Dataset<Row> createTable(String tableName, String source, StructType schema, String description, java.util.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)description
- (undocumented)options
- (undocumented)public abstract Dataset<Row> createTable(String tableName, String source, StructType schema, String description, scala.collection.immutable.Map<String,String> options)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in
the current database.source
- (undocumented)schema
- (undocumented)description
- (undocumented)options
- (undocumented)public abstract String currentCatalog()
public abstract String currentDatabase()
public abstract boolean databaseExists(String dbName)
dbName
- (undocumented)public abstract boolean dropGlobalTempView(String viewName)
Global temporary view is cross-session. Its lifetime is the lifetime of the Spark application,
i.e. it will be automatically dropped when the application terminates. It's tied to a system
preserved database global_temp
, and we must use the qualified name to refer a global temp
view, e.g. SELECT * FROM global_temp.view1
.
viewName
- the unqualified name of the temporary view to be dropped.public abstract boolean dropTempView(String viewName)
Local temporary view is session-scoped. Its lifetime is the lifetime of the session that
created it, i.e. it will be automatically dropped when the session terminates. It's not
tied to any databases, i.e. we can't use db1.view1
to reference a local temporary view.
Note that, the return type of this method was Unit in Spark 2.0, but changed to Boolean in Spark 2.1.
viewName
- the name of the temporary view to be dropped.public abstract boolean functionExists(String functionName)
functionName
- is either a qualified or unqualified name that designates a function. It
follows the same resolution rule with SQL: search for built-in/temp
functions first then functions in the current database (namespace).public abstract boolean functionExists(String dbName, String functionName)
To check existence of functions in other catalogs, please use functionExists(functionName)
with qualified function name instead.
dbName
- is an unqualified name that designates a database.functionName
- is an unqualified name that designates a function.public abstract Database getDatabase(String dbName) throws AnalysisException
dbName
- (undocumented)AnalysisException
public abstract Function getFunction(String functionName) throws AnalysisException
functionName
- is either a qualified or unqualified name that designates a function. It
follows the same resolution rule with SQL: search for built-in/temp
functions first then functions in the current database (namespace).AnalysisException
public abstract Function getFunction(String dbName, String functionName) throws AnalysisException
To get functions in other catalogs, please use getFunction(functionName)
with qualified
function name instead.
dbName
- is an unqualified name that designates a database.functionName
- is an unqualified name that designates a function in the specified databaseAnalysisException
public abstract Table getTable(String tableName) throws AnalysisException
tableName
- is either a qualified or unqualified name that designates a table/view. It
follows the same resolution rule with SQL: search for temp views first then
table/views in the current database (namespace).AnalysisException
public abstract Table getTable(String dbName, String tableName) throws AnalysisException
To get table/view in other catalogs, please use getTable(tableName)
with qualified table/view
name instead.
dbName
- (undocumented)tableName
- (undocumented)AnalysisException
public abstract boolean isCached(String tableName)
tableName
- is either a qualified or unqualified name that designates a table/view.
If no database identifier is provided, it refers to a temporary view or
a table/view in the current database.public abstract Dataset<CatalogMetadata> listCatalogs()
public abstract Dataset<CatalogMetadata> listCatalogs(String pattern)
pattern
- (undocumented)public abstract Dataset<Column> listColumns(String tableName) throws AnalysisException
tableName
- is either a qualified or unqualified name that designates a table/view. It
follows the same resolution rule with SQL: search for temp views first then
table/views in the current database (namespace).AnalysisException
public abstract Dataset<Column> listColumns(String dbName, String tableName) throws AnalysisException
To list columns for table/view in other catalogs, please use listColumns(tableName)
with
qualified table/view name instead.
dbName
- is an unqualified name that designates a database.tableName
- is an unqualified name that designates a table/view.AnalysisException
public abstract Dataset<Database> listDatabases()
public abstract Dataset<Database> listDatabases(String pattern)
pattern
- (undocumented)public abstract Dataset<Function> listFunctions()
public abstract Dataset<Function> listFunctions(String dbName) throws AnalysisException
dbName
- (undocumented)AnalysisException
public abstract Dataset<Function> listFunctions(String dbName, String pattern) throws AnalysisException
dbName
- (undocumented)pattern
- (undocumented)AnalysisException
public abstract Dataset<Table> listTables()
public abstract Dataset<Table> listTables(String dbName) throws AnalysisException
dbName
- (undocumented)AnalysisException
public abstract Dataset<Table> listTables(String dbName, String pattern) throws AnalysisException
dbName
- (undocumented)pattern
- (undocumented)AnalysisException
public abstract void recoverPartitions(String tableName)
tableName
- is either a qualified or unqualified name that designates a table.
If no database identifier is provided, it refers to a table in the
current database.public abstract void refreshByPath(String path)
Dataset
that contains the given data source path. Path matching is by checking for sub-directories,
i.e. "/" would invalidate everything that is cached and "/test/parent" would invalidate
everything that is a subdirectory of "/test/parent".
path
- (undocumented)public abstract void refreshTable(String tableName)
If this table is cached as an InMemoryRelation, drop the original cached version and make the new version cached lazily.
tableName
- is either a qualified or unqualified name that designates a table/view.
If no database identifier is provided, it refers to a temporary view or
a table/view in the current database.public abstract void setCurrentCatalog(String catalogName)
catalogName
- (undocumented)public abstract void setCurrentDatabase(String dbName)
dbName
- (undocumented)public abstract boolean tableExists(String tableName)
tableName
- is either a qualified or unqualified name that designates a table/view. It
follows the same resolution rule with SQL: search for temp views first then
table/views in the current database (namespace).public abstract boolean tableExists(String dbName, String tableName)
To check existence of table/view in other catalogs, please use tableExists(tableName)
with
qualified table/view name instead.
dbName
- is an unqualified name that designates a database.tableName
- is an unqualified name that designates a table.public abstract void uncacheTable(String tableName)
tableName
- is either a qualified or unqualified name that designates a table/view.
If no database identifier is provided, it refers to a temporary view or
a table/view in the current database.