Package org.apache.spark

Class ContextAwareIterator<T>

Object
org.apache.spark.ContextAwareIterator<T>
All Implemented Interfaces:
scala.collection.GenTraversableOnce<T>, scala.collection.Iterator<T>, scala.collection.TraversableOnce<T>
public class ContextAwareIterator<T> extends Object implements scala.collection.Iterator<T>
:: DeveloperApi :: A TaskContext aware iterator.

As the Python evaluation consumes the parent iterator in a separate thread, it could consume more data from the parent even after the task ends and the parent is closed. If an off-heap access exists in the parent iterator, it could cause segmentation fault which crashes the executor. Thus, we should use ContextAwareIterator to stop consuming after the task ends.

Since:
3.1.0

  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.collection.Iterator

    scala.collection.Iterator.GroupedIterator<B extends Object>, scala.collection.Iterator.SliceIterator<A extends Object>

    Nested classes/interfaces inherited from interface scala.collection.TraversableOnce

    scala.collection.TraversableOnce.BufferedCanBuildFrom<A extends Object,CC extends scala.collection.TraversableOnce<Object>>, scala.collection.TraversableOnce.FlattenOps<A extends Object>, scala.collection.TraversableOnce.ForceImplicitAmbiguity, scala.collection.TraversableOnce.MonadOps<A extends Object>, scala.collection.TraversableOnce.OnceCanBuildFrom<A extends Object>

  • Constructor Summary

    Constructors
    Constructor
    Description
    ContextAwareIterator(TaskContext context, scala.collection.Iterator<T> delegate)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    TaskContext
    context()
     
    scala.collection.Iterator<T>
    delegate()
     
    boolean
    hasNext()
     
    T
    next()
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface scala.collection.GenTraversableOnce

    sizeHintIfCheap

    Methods inherited from interface scala.collection.Iterator

    $plus$plus, buffered, collect, contains, copyToArray, corresponds, drop, dropWhile, duplicate, exists, filter, filterNot, find, flatMap, forall, foreach, grouped, hasDefiniteSize, indexOf, indexOf, indexWhere, indexWhere, isEmpty, isTraversableAgain, length, map, padTo, partition, patch, sameElements, scanLeft, scanRight, seq, slice, sliceIterator, sliding, sliding$default$2, span, take, takeWhile, toIterator, toStream, toString, toTraversable, withFilter, zip, zipAll, zipWithIndex

    Methods inherited from interface scala.collection.TraversableOnce

    $colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToBuffer, count, fold, foldLeft, foldRight, max, maxBy, min, minBy, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, size, sum, to, toArray, toBuffer, toIndexedSeq, toIterable, toList, toMap, toSeq, toSet, toVector

  • Constructor Details

    • ContextAwareIterator

      public ContextAwareIterator(TaskContext context, scala.collection.Iterator<T> delegate)

  • Method Details

    • context

      public TaskContext context()

    • delegate

      public scala.collection.Iterator<T> delegate()

    • hasNext

      public boolean hasNext()
      Specified by:
      hasNext in interface scala.collection.Iterator<T>

    • next

      public T next()
      Specified by:
      next in interface scala.collection.Iterator<T>