Class ShuffledRDD<K,V,C>

Object
org.apache.spark.rdd.RDD<scala.Tuple2<K,C>>
org.apache.spark.rdd.ShuffledRDD<K,V,C>
Type Parameters:
K - the key class.
V - the value class.
C - the combiner class.
All Implemented Interfaces:
Serializable, org.apache.spark.internal.Logging

public class ShuffledRDD<K,V,C> extends RDD<scala.Tuple2<K,C>>
:: DeveloperApi :: The resulting RDD from a shuffle (e.g. repartitioning of data). param: prev the parent RDD. param: part the partitioner used to partition the RDD
See Also:
  • Constructor Details

    • ShuffledRDD

      public ShuffledRDD(RDD<? extends scala.Product2<K,V>> prev, Partitioner part, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2, scala.reflect.ClassTag<C> evidence$3)
  • Method Details

    • clearDependencies

      public void clearDependencies()
    • compute

      public scala.collection.Iterator<scala.Tuple2<K,C>> compute(Partition split, TaskContext context)
      Description copied from class: RDD
      :: DeveloperApi :: Implemented by subclasses to compute a given partition.
      Specified by:
      compute in class RDD<scala.Tuple2<K,C>>
      Parameters:
      split - (undocumented)
      context - (undocumented)
      Returns:
      (undocumented)
    • getDependencies

      public scala.collection.immutable.Seq<Dependency<?>> getDependencies()
    • getPartitions

      public Partition[] getPartitions()
    • partitioner

      public scala.Some<Partitioner> partitioner()
      Description copied from class: RDD
      Optionally overridden by subclasses to specify how they are partitioned.
      Overrides:
      partitioner in class RDD<scala.Tuple2<K,C>>
    • prev

      public RDD<? extends scala.Product2<K,V>> prev()
    • setAggregator

      public ShuffledRDD<K,V,C> setAggregator(Aggregator<K,V,C> aggregator)
      Set aggregator for RDD's shuffle.
    • setKeyOrdering

      public ShuffledRDD<K,V,C> setKeyOrdering(scala.math.Ordering<K> keyOrdering)
      Set key ordering for RDD's shuffle.
    • setMapSideCombine

      public ShuffledRDD<K,V,C> setMapSideCombine(boolean mapSideCombine)
      Set mapSideCombine flag for RDD's shuffle.
    • setSerializer

      public ShuffledRDD<K,V,C> setSerializer(Serializer serializer)
      Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer)