org.apache.spark.sql.connector.read.partitioning

Interface Partitioning

@Evolving
public interface Partitioning

An interface to represent the output data partitioning for a data source, which is returned by SupportsReportPartitioning.outputPartitioning(). Note that this should work like a snapshot. Once created, it should be deterministic and always report the same number of partitions and the same "satisfy" result for a certain distribution.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
int	numPartitions() Returns the number of partitions(i.e., InputPartitions) the data source outputs.
boolean	satisfy(Distribution distribution) Returns true if this partitioning can satisfy the given distribution, which means Spark does not need to shuffle the output data of this data source for some certain operations.

Method Detail

numPartitions

int numPartitions()

Returns the number of partitions(i.e., InputPartitions) the data source outputs.

satisfy

boolean satisfy(Distribution distribution)

Returns true if this partitioning can satisfy the given distribution, which means Spark does not need to shuffle the output data of this data source for some certain operations. Note that, Spark may add new concrete implementations of Distribution in new releases. This method should be aware of it and always return false for unrecognized distributions. It's recommended to check every Spark new release and support new distributions if possible, to avoid shuffle at Spark side for more cases.