org.apache.spark.shuffle.api

Interface ShuffleMapOutputWriter

@Private
public interface ShuffleMapOutputWriter

:: Private :: A top-level writer that returns child writers for persisting the output of a map task, and then commits all of the writes as one atomic operation.

Since:

3.0.0

Method Summary

Modifier and Type	Method and Description
void	abort(Throwable error) Abort all of the writes done by any writers returned by getPartitionWriter(int).
long[]	commitAllPartitions() Commits the writes done by all partition writers returned by all calls to this object's getPartitionWriter(int), and returns the number of bytes written for each partition.
ShufflePartitionWriter	getPartitionWriter(int reducePartitionId) Creates a writer that can open an output stream to persist bytes targeted for a given reduce partition id.

Method Detail

getPartitionWriter

ShufflePartitionWriter getPartitionWriter(int reducePartitionId)
                                   throws java.io.IOException

Creates a writer that can open an output stream to persist bytes targeted for a given reduce partition id.

The chunk corresponds to bytes in the given reduce partition. This will not be called twice for the same partition within any given map task. The partition identifier will be in the range of precisely 0 (inclusive) to numPartitions (exclusive), where numPartitions was provided upon the creation of this map output writer via ShuffleExecutorComponents.createMapOutputWriter(int, long, int).

Calls to this method will be invoked with monotonically increasing reducePartitionIds; each call to this method will be called with a reducePartitionId that is strictly greater than the reducePartitionIds given to any previous call to this method. This method is not guaranteed to be called for every partition id in the above described range. In particular, no guarantees are made as to whether or not this method will be called for empty partitions.

Throws:

java.io.IOException

commitAllPartitions

long[] commitAllPartitions()
                    throws java.io.IOException

Commits the writes done by all partition writers returned by all calls to this object's getPartitionWriter(int), and returns the number of bytes written for each partition.

This should ensure that the writes conducted by this module's partition writers are available to downstream reduce tasks. If this method throws any exception, this module's abort(Throwable) method will be invoked before propagating the exception.

This can also close any resources and clean up temporary state if necessary.

The returned array should contain, for each partition from (0) to (numPartitions - 1), the number of bytes written by the partition writer for that partition id.

Throws:

java.io.IOException

abort

void abort(Throwable error)
    throws java.io.IOException

Abort all of the writes done by any writers returned by getPartitionWriter(int).

This should invalidate the results of writing bytes. This can also close any resources and clean up temporary state if necessary.

Throws:

java.io.IOException