Package org.apache.spark

Class PipelinedShuffleDependency<K,V,C>

Object
org.apache.spark.Dependency<scala.Product2<K,V>>
org.apache.spark.ShuffleDependency<K,V,C>
org.apache.spark.PipelinedShuffleDependency<K,V,C>
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging
public class PipelinedShuffleDependency<K,V,C> extends ShuffleDependency<K,V,C>
:: DeveloperApi :: A ShuffleDependency whose output can be read incrementally: a consumer stage may begin reading the shuffle output while the producer stage is still running, rather than waiting for the producer's full, materialized output.

This is a subtype of ShuffleDependency -- and thus, like it, a first-class dependency kind alongside NarrowDependency under Dependency. It is intended to be the marker the DAGScheduler will use to decide that the producer and consumer stages connected by this edge may run concurrently (a "pipelined group"), and that the shuffle layer should serve this shuffle with an incremental shuffle implementation. A plain ShuffleDependency keeps the existing semantics: its output is fully materialized before any consumer reads it.

Two behaviors are active from construction, driven by this type: shuffle registration routes to the incremental (pipelined) shuffle manager via SparkEnv.shuffleManagerFor (a plain ShuffleDependency goes to the blocking manager), and push-based shuffle merge is unconditionally disabled (setShuffleMergeAllowed(false); see below) because merge exposes output only after a post-completion finalize step and would register merge results for a transient shuffle. Beyond those, the concurrent scheduling behavior is added separately by the DAGScheduler components that match on this type; code that only matches the parent ShuffleDependency still treats it as an ordinary (materialized) shuffle for everything else.

The name is *pipelined* rather than *streaming*: reading producer output as it is produced is a general execution capability (software-pipelining of dependent stages), not specific to streaming. Streaming / real-time mode is the first caller, but nothing here is streaming-specific.

Note: the parent's checksumMismatchFullRetryEnabled / checksumMismatchQueryLevelRollbackEnabled parameters are intentionally not exposed here, so they stay at their false defaults for a pipelined shuffle. Their checksum retry / query-level rollback recomputes and re-runs succeeding stages after a mismatch; in a pipelined group any failure aborts the whole group and the caller reruns from scratch, so that stage-level recompute never fires -- the mechanism is moot by construction (it is also incompatible with a consumer that has already read the output incrementally). Leaving the params unset keeps the idiom unreachable.

See Also:

  • Constructor Details

    • PipelinedShuffleDependency

      public PipelinedShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, org.apache.spark.shuffle.checksum.RowBasedChecksum[] rowBasedChecksums, scala.reflect.ClassTag<K> evidence$7, scala.reflect.ClassTag<V> evidence$8, scala.reflect.ClassTag<C> evidence$9)