trait SupportsCatalogOptions extends TableProvider
An interface, which TableProviders can implement, to support table existence checks and creation
through a catalog, without having to use table identifiers. For example, when file based data
sources use the DataFrameWriter.save(path) method, the option path can translate to a
PathIdentifier. A catalog can then use this PathIdentifier to check the existence of a table, or
whether a table can be created at a given directory.
- Annotations
- @Evolving()
- Source
- SupportsCatalogOptions.java
- Since
3.0.0
- Alphabetic
- By Inheritance
- SupportsCatalogOptions
- TableProvider
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def extractIdentifier(options: CaseInsensitiveStringMap): Identifier
Return a
Identifierinstance that can identify a table for a DataSource given DataFrame[Reader|Writer] options.Return a
Identifierinstance that can identify a table for a DataSource given DataFrame[Reader|Writer] options.- options
the user-specified options that can identify a table, e.g. file path, Kafka topic name, etc. It's an immutable case-insensitive string-to-string map.
- abstract def getTable(schema: StructType, partitioning: Array[Transform], properties: Map[String, String]): Table
Return a
Tableinstance with the specified table schema, partitioning and properties to do read/write.Return a
Tableinstance with the specified table schema, partitioning and properties to do read/write. The returned table should report the same schema and partitioning with the specified ones, or Spark may fail the operation.- schema
The specified table schema.
- partitioning
The specified table partitioning.
- properties
The specified table properties. It's case preserving (contains exactly what users specified) and implementations are free to use it case sensitively or insensitively. It should be able to identify a table, e.g. file path, Kafka topic name, etc.
- Definition Classes
- TableProvider
- abstract def inferSchema(options: CaseInsensitiveStringMap): StructType
Infer the schema of the table identified by the given options.
Infer the schema of the table identified by the given options.
- options
an immutable case-insensitive string-to-string map that can identify a table, e.g. file path, Kafka topic name, etc.
- Definition Classes
- TableProvider
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def extractCatalog(options: CaseInsensitiveStringMap): String
Return the name of a catalog that can be used to check the existence of, load, and create a table for this DataSource given the identifier that will be extracted by
extractIdentifier.Return the name of a catalog that can be used to check the existence of, load, and create a table for this DataSource given the identifier that will be extracted by
extractIdentifier. Anullvalue can be used to defer to the V2SessionCatalog.- options
the user-specified options that can identify a table, e.g. file path, Kafka topic name, etc. It's an immutable case-insensitive string-to-string map.
- def extractTimeTravelTimestamp(options: CaseInsensitiveStringMap): Optional[String]
Extracts the timestamp string for time travel from the given options.
- def extractTimeTravelVersion(options: CaseInsensitiveStringMap): Optional[String]
Extracts the version string for time travel from the given options.
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def inferPartitioning(options: CaseInsensitiveStringMap): Array[Transform]
Infer the partitioning of the table identified by the given options.
Infer the partitioning of the table identified by the given options.
By default this method returns empty partitioning, please override it if this source support partitioning.
- options
an immutable case-insensitive string-to-string map that can identify a table, e.g. file path, Kafka topic name, etc.
- Definition Classes
- TableProvider
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def supportsExternalMetadata(): Boolean
Returns true if the source has the ability of accepting external table metadata when getting tables.
Returns true if the source has the ability of accepting external table metadata when getting tables. The external table metadata includes:
- For table reader: user-specified schema from
DataFrameReader/DataStreamReaderand schema/partitioning stored in Spark catalog. - For table writer: the schema of the input
DataframeofDataframeWriter/DataStreamWriter.
By default this method returns false, which means the schema and partitioning passed to
Transform[], Map)are from the infer methods. Please override it if this source has expensive schema/partitioning inference and wants external table metadata to avoid inference.- Definition Classes
- TableProvider
- For table reader: user-specified schema from
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)