Package org.apache.spark.scheduler

Interface MapStatus

All Superinterfaces:
ShuffleOutputStatus
public interface MapStatus extends ShuffleOutputStatus
Result returned by a ShuffleMapTask to a scheduler. Includes the block manager address that the task has shuffle files stored on as well as the sizes of outputs for each reducer, for passing on to the reduce tasks.

  • Method Summary

    Modifier and Type
    Method
    Description
    long
    checksumValue()
    The checksum value of this shuffle map task, which can be used to evaluate whether the output data has changed across different map task retries.
    long
    getSizeForBlock(int reduceId)
    Estimated size for the reduce block, in bytes.
    BlockManagerId
    location()
    Location where this task output is.
    long
    mapId()
    The unique ID of this shuffle map task, if spark.shuffle.useOldFetchProtocol enabled we use partitionId of the task or taskContext.taskAttemptId is used.
    void
    updateLocation(BlockManagerId newLoc)
     

  • Method Details

    • location

      BlockManagerId location()
      Location where this task output is.

    • updateLocation

      void updateLocation(BlockManagerId newLoc)

    • getSizeForBlock

      long getSizeForBlock(int reduceId)
      Estimated size for the reduce block, in bytes.

      If a block is non-empty, then this method MUST return a non-zero size. This invariant is necessary for correctness, since block fetchers are allowed to skip zero-size blocks.

      Parameters:
      reduceId - (undocumented)
      Returns:
      (undocumented)

    • mapId

      long mapId()
      The unique ID of this shuffle map task, if spark.shuffle.useOldFetchProtocol enabled we use partitionId of the task or taskContext.taskAttemptId is used.
      Returns:
      (undocumented)

    • checksumValue

      long checksumValue()
      The checksum value of this shuffle map task, which can be used to evaluate whether the output data has changed across different map task retries.
      Returns:
      (undocumented)