org.apache.spark.sql.connector.read

Interface InputPartition

All Superinterfaces:

java.io.Serializable

All Known Subinterfaces:

HasPartitionKey

@Evolving
public interface InputPartition
extends java.io.Serializable

A serializable representation of an input partition returned by Batch.planInputPartitions() and the corresponding ones in streaming .

Note that InputPartition will be serialized and sent to executors, then PartitionReader will be created by PartitionReaderFactory.createReader(InputPartition) or PartitionReaderFactory.createColumnarReader(InputPartition) on executors to do the actual reading. So InputPartition must be serializable while PartitionReader doesn't need to be.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
default String[]	preferredLocations() The preferred locations where the input partition reader returned by this partition can run faster, but Spark does not guarantee to run the input partition reader on these locations.

Method Detail

preferredLocations

default String[] preferredLocations()

The preferred locations where the input partition reader returned by this partition can run faster, but Spark does not guarantee to run the input partition reader on these locations. The implementations should make sure that it can be run on any location. The location is a string representing the host name.

Note that if a host name cannot be recognized by Spark, it will be ignored as it was not in the returned locations. The default return value is empty string array, which means this input partition's reader has no location preference.

If this method fails (by throwing an exception), the action will fail and no Spark job will be submitted.