Package org.apache.spark.sql.pipelines.graph

Class Table

Object
org.apache.spark.sql.pipelines.graph.Table
All Implemented Interfaces:
Serializable, Dataset, GraphElement, Input, Output, TableInput, scala.Equals, scala.Product
public class Table extends Object implements TableInput, Dataset, scala.Product, Serializable
A table representing a materialized dataset in a DataflowGraph.

param: identifier The identifier of this table within the graph. param: specifiedSchema The user-specified schema for this table. param: partitionCols What columns the table should be partitioned by when materialized. param: normalizedPath Normalized storage location for the table based on the user-specified table path (if not defined, we will normalize a managed storage path for it). param: properties Table Properties to set in table metadata. param: comment User-specified comment that can be placed on the table. param: isStreamingTable if the table is a streaming table, as defined by the source code.

See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    Table(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String,String> properties, scala.Option<String> comment, QueryOrigin origin, boolean isStreamingTable, scala.Option<String> format)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    abstract static R
    apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9)
     
    scala.Option<String>
    comment()
     
    DatasetType
    datasetType()
    Get the DatasetType of the table
    scala.Option<String>
    format()
     
    org.apache.spark.sql.catalyst.TableIdentifier
    identifier()
    Returns the unique identifier for this GraphElement.
    boolean
    isStreamingTable()
     
    org.apache.spark.sql.classic.Dataset<Row>
    load(InputReadOptions readOptions)
    Returns a DataFrame that is a result of loading data from this Input.
    scala.Option<String>
    normalizedPath()
    Normalized storage location used for storing materializations for this Output.
    QueryOrigin
    origin()
    Contains provenance to tie back this GraphElement to the user code that defined it.
    scala.Option<scala.collection.immutable.Seq<String>>
    partitionCols()
     
    String
    path()
    Returns the normalized storage location to this Table.
    scala.collection.immutable.Map<String,String>
    properties()
     
    scala.Option<StructType>
    specifiedSchema()
    The user-specified schema for this table.
    static String
    toString()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.pipelines.graph.Dataset

    normalized

    Methods inherited from interface scala.Equals

    canEqual, equals

    Methods inherited from interface org.apache.spark.sql.pipelines.graph.GraphElement

    displayName, spark

    Methods inherited from interface scala.Product

    productArity, productElement, productElementName, productElementNames, productIterator, productPrefix

  • Constructor Details

    • Table

      public Table(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String,String> properties, scala.Option<String> comment, QueryOrigin origin, boolean isStreamingTable, scala.Option<String> format)

  • Method Details

    • apply

      public abstract static R apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9)

    • toString

      public static String toString()

    • identifier

      public org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Description copied from interface: GraphElement
      Returns the unique identifier for this GraphElement.
      Specified by:
      identifier in interface GraphElement

    • specifiedSchema

      public scala.Option<StructType> specifiedSchema()
      Description copied from interface: TableInput
      The user-specified schema for this table.
      Specified by:
      specifiedSchema in interface TableInput

    • partitionCols

      public scala.Option<scala.collection.immutable.Seq<String>> partitionCols()

    • normalizedPath

      public scala.Option<String> normalizedPath()
      Description copied from interface: Dataset
      Normalized storage location used for storing materializations for this Output. If None, it means this Output has not been normalized yet.
      Specified by:
      normalizedPath in interface Dataset
      Returns:
      (undocumented)

    • properties

      public scala.collection.immutable.Map<String,String> properties()

    • comment

      public scala.Option<String> comment()

    • origin

      public QueryOrigin origin()
      Description copied from interface: GraphElement
      Contains provenance to tie back this GraphElement to the user code that defined it.

      This must be set when a GraphElement is directly created by some user code. Subsequently, this initial origin must be propagated as is without modification. If this GraphElement is copied or converted to a different type, then this origin must be copied as is.

      Specified by:
      origin in interface GraphElement
      Returns:
      (undocumented)

    • isStreamingTable

      public boolean isStreamingTable()

    • format

      public scala.Option<String> format()

    • load

      public org.apache.spark.sql.classic.Dataset<Row> load(InputReadOptions readOptions)
      Description copied from interface: Input
      Returns a DataFrame that is a result of loading data from this Input.
      Specified by:
      load in interface Input
      Parameters:
      readOptions - Type of input. Used to determine streaming/batch
      Returns:
      Streaming or batch DataFrame of this Input's data.

    • path

      public String path()
      Returns the normalized storage location to this Table.
      Specified by:
      path in interface Dataset
      Returns:
      (undocumented)

    • datasetType

      public DatasetType datasetType()
      Get the DatasetType of the table
      Returns:
      (undocumented)