Package org.apache.spark.sql.connector.catalog

Interface MetadataColumn

@Evolving public interface MetadataColumn
Interface for a metadata column.

A metadata column can expose additional metadata about a row. For example, rows from Kafka can use metadata columns to expose a message's topic, partition number, and offset.

A metadata column could also be the result of a transform applied to a value in the row. For example, a partition value produced by bucket(id, 16) could be exposed by a metadata column. In this case, transform() should return a non-null Transform that produced the metadata column's values.

Since:
3.1.0

  • Field Details

    • PRESERVE_ON_DELETE

      static final String PRESERVE_ON_DELETE
      Indicates whether a row-level operation should preserve the value of the metadata column for deleted rows. If set to true, the metadata value will be retained and passed back to the writer. If false, the metadata value will be replaced with null.

      This flag applies only to row-level operations working with deltas of rows. Group-based operations handle deletes by discarding matching records.

      Since:
      4.0.0
      See Also:

    • PRESERVE_ON_DELETE_DEFAULT

      static final boolean PRESERVE_ON_DELETE_DEFAULT
      See Also:

    • PRESERVE_ON_UPDATE

      static final String PRESERVE_ON_UPDATE
      Indicates whether a row-level operation should preserve the value of the metadata column for updated rows. If set to true, the metadata value will be retained and passed back to the writer. If false, the metadata value will be replaced with null.

      This flag applies to both group-based and delta-based row-level operations.

      Since:
      4.0.0
      See Also:

    • PRESERVE_ON_UPDATE_DEFAULT

      static final boolean PRESERVE_ON_UPDATE_DEFAULT
      See Also:

    • PRESERVE_ON_REINSERT

      static final String PRESERVE_ON_REINSERT
      Indicates whether a row-level operation should preserve the value of the metadata column for reinserted rows generated by splitting updates into deletes and inserts. If true, the metadata value will be retained and passed back to the writer. If false, the metadata value will be replaced with null.

      This flag applies only to row-level operations working with deltas of rows. Group-based operations do not represent updates as deletes and inserts.

      Since:
      4.0.0
      See Also:

    • PRESERVE_ON_REINSERT_DEFAULT

      static final boolean PRESERVE_ON_REINSERT_DEFAULT
      See Also:

  • Method Details

    • name

      String name()
      The name of this metadata column.
      Returns:
      a String name

    • dataType

      DataType dataType()
      The data type of values in this metadata column.
      Returns:
      a DataType

    • isNullable

      default boolean isNullable()
      Returns:
      whether values produced by this metadata column may be null

    • comment

      default String comment()
      Documentation for this metadata column, or null.
      Returns:
      a documentation String

    • transform

      default Transform transform()
      The Transform used to produce this metadata column from data rows, or null.
      Returns:
      a Transform used to produce the column's values, or null if there isn't one

    • metadataInJSON

      default String metadataInJSON()
      Returns the column metadata in JSON format.
      Since:
      4.0.0