Package org.apache.spark

Class ShuffleDependency<K,V,C>

Object
org.apache.spark.Dependency<scala.Product2<K,V>>
org.apache.spark.ShuffleDependency<K,V,C>
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging
public class ShuffleDependency<K,V,C> extends Dependency<scala.Product2<K,V>> implements org.apache.spark.internal.Logging
See Also:

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, org.apache.spark.shuffle.checksum.RowBasedChecksum[] rowBasedChecksums, boolean checksumMismatchFullRetryEnabled, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)
     
    ShuffleDependency(RDD<? extends scala.Product2<K,V>> rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, scala.reflect.ClassTag<K> evidence$4, scala.reflect.ClassTag<V> evidence$5, scala.reflect.ClassTag<C> evidence$6)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    scala.Option<Aggregator<K,V,C>>
    aggregator()
     
    boolean
    checksumMismatchFullRetryEnabled()
     
    scala.collection.immutable.Seq<BlockManagerId>
    getMergerLocs()
     
    scala.Option<scala.math.Ordering<K>>
    keyOrdering()
     
    boolean
    mapSideCombine()
     
    void
    newShuffleMergeState()
     
    Partitioner
    partitioner()
     
    RDD<scala.Product2<K,V>>
    rdd()
     
    org.apache.spark.shuffle.checksum.RowBasedChecksum[]
    rowBasedChecksums()
     
    Serializer
    serializer()
     
    void
    setMergerLocs(scala.collection.immutable.Seq<BlockManagerId> mergerLocs)
     
    org.apache.spark.shuffle.ShuffleHandle
    shuffleHandle()
     
    int
    shuffleId()
     
    boolean
    shuffleMergeAllowed()
     
    boolean
    shuffleMergeEnabled()
     
    boolean
    shuffleMergeFinalized()
    Returns true if push-based shuffle is disabled or if the shuffle merge for this shuffle is finalized.
    int
    shuffleMergeId()
    shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.
    org.apache.spark.shuffle.ShuffleWriteProcessor
    shuffleWriterProcessor()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

    • ShuffleDependency

      public ShuffleDependency(RDD<? extends scala.Product2<K,V>> _rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, org.apache.spark.shuffle.checksum.RowBasedChecksum[] rowBasedChecksums, boolean checksumMismatchFullRetryEnabled, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)

    • ShuffleDependency

      public ShuffleDependency(RDD<? extends scala.Product2<K,V>> rdd, Partitioner partitioner, Serializer serializer, scala.Option<scala.math.Ordering<K>> keyOrdering, scala.Option<Aggregator<K,V,C>> aggregator, boolean mapSideCombine, org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor, scala.reflect.ClassTag<K> evidence$4, scala.reflect.ClassTag<V> evidence$5, scala.reflect.ClassTag<C> evidence$6)

  • Method Details

    • partitioner

      public Partitioner partitioner()

    • serializer

      public Serializer serializer()

    • keyOrdering

      public scala.Option<scala.math.Ordering<K>> keyOrdering()

    • aggregator

      public scala.Option<Aggregator<K,V,C>> aggregator()

    • mapSideCombine

      public boolean mapSideCombine()

    • shuffleWriterProcessor

      public org.apache.spark.shuffle.ShuffleWriteProcessor shuffleWriterProcessor()

    • rowBasedChecksums

      public org.apache.spark.shuffle.checksum.RowBasedChecksum[] rowBasedChecksums()

    • checksumMismatchFullRetryEnabled

      public boolean checksumMismatchFullRetryEnabled()

    • rdd

      public RDD<scala.Product2<K,V>> rdd()
      Specified by:
      rdd in class Dependency<scala.Product2<K,V>>

    • shuffleId

      public int shuffleId()

    • shuffleHandle

      public org.apache.spark.shuffle.ShuffleHandle shuffleHandle()

    • shuffleMergeEnabled

      public boolean shuffleMergeEnabled()

    • shuffleMergeAllowed

      public boolean shuffleMergeAllowed()

    • shuffleMergeId

      public int shuffleMergeId()
      shuffleMergeId is used to uniquely identify merging process of shuffle by an indeterminate stage attempt.
      Returns:
      (undocumented)

    • setMergerLocs

      public void setMergerLocs(scala.collection.immutable.Seq<BlockManagerId> mergerLocs)

    • getMergerLocs

      public scala.collection.immutable.Seq<BlockManagerId> getMergerLocs()

    • shuffleMergeFinalized

      public boolean shuffleMergeFinalized()
      Returns true if push-based shuffle is disabled or if the shuffle merge for this shuffle is finalized.
      Returns:
      (undocumented)

    • newShuffleMergeState

      public void newShuffleMergeState()