Object
org.apache.spark.sql.pipelines.graph.Table
All Implemented Interfaces:
Serializable, GraphElement, Input, Output, TableInput, scala.Equals, scala.Product

public class Table extends Object implements TableInput, Output, scala.Product, Serializable
A table representing a materialized dataset in a DataflowGraph.

param: identifier The identifier of this table within the graph. param: specifiedSchema The user-specified schema for this table. param: partitionCols What columns the table should be partitioned by when materialized. param: normalizedPath Normalized storage location for the table based on the user-specified table path (if not defined, we will normalize a managed storage path for it). param: properties Table Properties to set in table metadata. param: comment User-specified comment that can be placed on the table. param: isStreamingTable if the table is a streaming table, as defined by the source code.

See Also:
  • Constructor Details

    • Table

      public Table(org.apache.spark.sql.catalyst.TableIdentifier identifier, scala.Option<StructType> specifiedSchema, scala.Option<scala.collection.immutable.Seq<String>> partitionCols, scala.Option<String> normalizedPath, scala.collection.immutable.Map<String,String> properties, scala.Option<String> comment, QueryOrigin baseOrigin, boolean isStreamingTable, scala.Option<String> format)
  • Method Details

    • apply

      public abstract static R apply(T1 v1, T2 v2, T3 v3, T4 v4, T5 v5, T6 v6, T7 v7, T8 v8, T9 v9)
    • toString

      public static String toString()
    • identifier

      public org.apache.spark.sql.catalyst.TableIdentifier identifier()
      Description copied from interface: GraphElement
      Returns the unique identifier for this GraphElement.
      Specified by:
      identifier in interface GraphElement
    • specifiedSchema

      public scala.Option<StructType> specifiedSchema()
      Description copied from interface: TableInput
      The user-specified schema for this table.
      Specified by:
      specifiedSchema in interface TableInput
    • partitionCols

      public scala.Option<scala.collection.immutable.Seq<String>> partitionCols()
    • normalizedPath

      public scala.Option<String> normalizedPath()
      Description copied from interface: Output
      Normalized storage location used for storing materializations for this Output. If None, it means this Output has not been normalized yet.
      Specified by:
      normalizedPath in interface Output
      Returns:
      (undocumented)
    • properties

      public scala.collection.immutable.Map<String,String> properties()
    • comment

      public scala.Option<String> comment()
    • baseOrigin

      public QueryOrigin baseOrigin()
    • isStreamingTable

      public boolean isStreamingTable()
    • format

      public scala.Option<String> format()
    • origin

      public QueryOrigin origin()
      Description copied from interface: GraphElement
      Contains provenance to tie back this GraphElement to the user code that defined it.

      This must be set when a GraphElement is directly created by some user code. Subsequently, this initial origin must be propagated as is without modification. If this GraphElement is copied or converted to a different type, then this origin must be copied as is.

      Specified by:
      origin in interface GraphElement
      Returns:
      (undocumented)
    • load

      public org.apache.spark.sql.classic.Dataset<Row> load(InputReadOptions readOptions)
      Description copied from interface: Input
      Returns a DataFrame that is a result of loading data from this Input.
      Specified by:
      load in interface Input
      Parameters:
      readOptions - Type of input. Used to determine streaming/batch
      Returns:
      Streaming or batch DataFrame of this Input's data.
    • path

      public String path()
      Returns the normalized storage location to this Table.
      Specified by:
      path in interface Output
      Returns:
      (undocumented)
    • datasetType

      public DatasetType datasetType()
      Get the DatasetType of the table
      Returns:
      (undocumented)