Package org.apache.spark.shuffle.api

Interface ShuffleExecutorComponents

@Private public interface ShuffleExecutorComponents
:: Private :: An interface for building shuffle support for Executors.
Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    ShuffleMapOutputWriter
    createMapOutputWriter(int shuffleId, long mapTaskId, int numPartitions)
    Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.
    default Optional<SingleSpillShuffleMapOutputWriter>
    createSingleFileMapOutputWriter(int shuffleId, long mapId)
    An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.
    void
    initializeExecutor(String appId, String execId, Map<String,String> extraConfigs)
    Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.

  • Method Details

    • initializeExecutor

      void initializeExecutor(String appId, String execId, Map<String,String> extraConfigs)
      Called once per executor to bootstrap this module with state that is specific to that executor, specifically the application ID and executor ID.
      Parameters:
      appId - The Spark application id
      execId - The unique identifier of the executor being initialized
      extraConfigs - Extra configs that were returned by ShuffleDriverComponents.initializeApplication()

    • createMapOutputWriter

      ShuffleMapOutputWriter createMapOutputWriter(int shuffleId, long mapTaskId, int numPartitions) throws IOException
      Called once per map task to create a writer that will be responsible for persisting all the partitioned bytes written by that map task.
      Parameters:
      shuffleId - Unique identifier for the shuffle the map task is a part of
      mapTaskId - An ID of the map task. The ID is unique within this Spark application.
      numPartitions - The number of partitions that will be written by the map task. Some of these partitions may be empty.
      Throws:
      IOException

    • createSingleFileMapOutputWriter

      default Optional<SingleSpillShuffleMapOutputWriter> createSingleFileMapOutputWriter(int shuffleId, long mapId) throws IOException
      An optional extension for creating a map output writer that can optimize the transfer of a single partition file, as the entire result of a map task, to the backing store.

      Most implementations should return the default Optional.empty() to indicate that they do not support this optimization. This primarily is for backwards-compatibility in preserving an optimization in the local disk shuffle storage implementation.

      Parameters:
      shuffleId - Unique identifier for the shuffle the map task is a part of
      mapId - An ID of the map task. The ID is unique within this Spark application.
      Throws:
      IOException