Class Table
- All Implemented Interfaces:
Serializable
,GraphElement
,Input
,Output
,TableInput
,scala.Equals
,scala.Product
DataflowGraph
.
param: identifier The identifier of this table within the graph. param: specifiedSchema The user-specified schema for this table. param: partitionCols What columns the table should be partitioned by when materialized. param: normalizedPath Normalized storage location for the table based on the user-specified table path (if not defined, we will normalize a managed storage path for it). param: properties Table Properties to set in table metadata. param: comment User-specified comment that can be placed on the table. param: isStreamingTable if the table is a streaming table, as defined by the source code.
- See Also:
-
Constructor Summary
ConstructorsConstructorDescriptionTable
(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String, String> properties, scala.Option<String> comment, QueryOrigin baseOrigin, boolean isStreamingTable, scala.Option<String> format) -
Method Summary
Modifier and TypeMethodDescriptionabstract static R
apply
(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9) scala.Option<String>
comment()
Get the DatasetType of the tablescala.Option<String>
format()
org.apache.spark.sql.catalyst.TableIdentifier
Returns the unique identifier for thisGraphElement
.boolean
org.apache.spark.sql.classic.Dataset<Row>
load
(InputReadOptions readOptions) Returns a DataFrame that is a result of loading data from thisInput
.scala.Option<String>
Normalized storage location used for storing materializations for thisOutput
.origin()
Contains provenance to tie back this GraphElement to the user code that defined it.scala.Option<scala.collection.immutable.Seq<String>>
path()
Returns the normalized storage location to thisTable
.scala.Option<StructType>
The user-specified schema for this table.static String
toString()
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface scala.Equals
canEqual, equals
Methods inherited from interface org.apache.spark.sql.pipelines.graph.GraphElement
displayName, spark
Methods inherited from interface org.apache.spark.sql.pipelines.graph.Output
normalized
Methods inherited from interface scala.Product
productArity, productElement, productElementName, productElementNames, productIterator, productPrefix
-
Constructor Details
-
Table
public Table(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String, String> properties, scala.Option<String> comment, QueryOrigin baseOrigin, boolean isStreamingTable, scala.Option<String> format)
-
-
Method Details
-
apply
public abstract static R apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9) -
toString
-
identifier
public org.apache.spark.sql.catalyst.TableIdentifier identifier()Description copied from interface:GraphElement
Returns the unique identifier for thisGraphElement
.- Specified by:
identifier
in interfaceGraphElement
-
specifiedSchema
Description copied from interface:TableInput
The user-specified schema for this table.- Specified by:
specifiedSchema
in interfaceTableInput
-
partitionCols
-
normalizedPath
Description copied from interface:Output
Normalized storage location used for storing materializations for thisOutput
. If None, it means thisOutput
has not been normalized yet.- Specified by:
normalizedPath
in interfaceOutput
- Returns:
- (undocumented)
-
properties
-
comment
-
baseOrigin
-
isStreamingTable
public boolean isStreamingTable() -
format
-
origin
Description copied from interface:GraphElement
Contains provenance to tie back this GraphElement to the user code that defined it.This must be set when a
GraphElement
is directly created by some user code. Subsequently, this initial origin must be propagated as is without modification. If thisGraphElement
is copied or converted to a different type, then this origin must be copied as is.- Specified by:
origin
in interfaceGraphElement
- Returns:
- (undocumented)
-
load
Description copied from interface:Input
Returns a DataFrame that is a result of loading data from thisInput
. -
path
Returns the normalized storage location to thisTable
. -
datasetType
Get the DatasetType of the table- Returns:
- (undocumented)
-