Packages

trait SupportsDelta extends RowLevelOperation

A mix-in interface for RowLevelOperation. Data sources can implement this interface to indicate they support handling deltas of rows.

Annotations
@Experimental()
Source
SupportsDelta.java
Since

3.4.0

Linear Supertypes
RowLevelOperation, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SupportsDelta
  2. RowLevelOperation
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Abstract Value Members

  1. abstract def command(): Command

    Returns the SQL command that is being performed.

    Returns the SQL command that is being performed.

    Definition Classes
    RowLevelOperation
  2. abstract def newScanBuilder(options: CaseInsensitiveStringMap): ScanBuilder

    Returns a ScanBuilder to configure a Scan for this row-level operation.

    Returns a ScanBuilder to configure a Scan for this row-level operation.

    Data sources fall into two categories: those that can handle a delta of rows and those that need to replace groups (e.g. partitions, files). Data sources that handle deltas allow Spark to quickly discard unchanged rows and have no requirements for input scans. Data sources that replace groups of rows can discard deleted rows but need to keep unchanged rows to be passed back into the source. This means that scans for such data sources must produce all rows in a group if any are returned. Some data sources will avoid pushing filters into files (file granularity), while others will avoid pruning files within a partition (partition granularity).

    For example, if a data source can only replace partitions, all rows from a partition must be returned by the scan, even if a filter can narrow the set of changes to a single file in the partition. Similarly, a data source that can swap individual files must produce all rows from files where at least one record must be changed, not just rows that must be changed.

    Data sources that replace groups of data (e.g. files, partitions) may prune entire groups using provided data source filters when building a scan for this row-level operation. However, such data skipping is limited as not all expressions can be converted into data source filters and some can only be evaluated by Spark (e.g. subqueries). Since rewriting groups is expensive, Spark allows group-based data sources to filter groups at runtime. The runtime filtering enables data sources to narrow down the scope of rewriting to only groups that must be rewritten. If the row-level operation scan implements SupportsRuntimeV2Filtering, Spark will execute a query at runtime to find which records match the row-level condition. The runtime group filter subquery will leverage a regular batch scan, which isn't required to produce all rows in a group if any are returned. The information about matching records will be passed back into the row-level operation scan, allowing data sources to discard groups that don't have to be rewritten.

    Definition Classes
    RowLevelOperation
  3. abstract def newWriteBuilder(info: LogicalWriteInfo): DeltaWriteBuilder

    Returns a WriteBuilder to configure a Write for this row-level operation.

    Returns a WriteBuilder to configure a Write for this row-level operation.

    Note that Spark will first configure the scan and then the write, allowing data sources to pass information from the scan to the write. For example, the scan can report which condition was used to read the data that may be needed by the write under certain isolation levels. Implementations may capture the built scan or required scan information and then use it while building the write.

    Definition Classes
    SupportsDeltaRowLevelOperation
    Annotations
    @Override()
  4. abstract def rowId(): Array[NamedReference]

    Returns the row ID column references that should be used for row equality.

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  6. def description(): String

    Returns the description associated with this row-level operation.

    Returns the description associated with this row-level operation.

    Definition Classes
    RowLevelOperation
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  15. def representUpdateAsDeleteAndInsert(): Boolean

    Controls whether to represent updates as deletes and inserts.

    Controls whether to represent updates as deletes and inserts.

    Data sources may choose to split updates into deletes and inserts to either better cluster and order the incoming delta of rows or to simplify the write process.

  16. def requiredMetadataAttributes(): Array[NamedReference]

    Returns metadata attributes that are required to perform this row-level operation.

    Returns metadata attributes that are required to perform this row-level operation.

    Data sources that can use this method to project metadata columns needed for writing the data back (e.g. metadata columns for grouping data).

    Definition Classes
    RowLevelOperation
  17. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  21. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from RowLevelOperation

Inherited from AnyRef

Inherited from Any

Ungrouped