Package org.apache.spark

Class ContextAwareIterator<T>

Object
org.apache.spark.ContextAwareIterator<T>
All Implemented Interfaces:
scala.collection.IterableOnce<T>, scala.collection.IterableOnceOps<T,scala.collection.Iterator,scala.collection.Iterator<T>>, scala.collection.Iterator<T>
public class ContextAwareIterator<T> extends Object implements scala.collection.Iterator<T>
Deprecated.
since 4.0.0 as its only usage for Python evaluation is now extinct
:: DeveloperApi :: A TaskContext aware iterator.

As the Python evaluation consumes the parent iterator in a separate thread, it could consume more data from the parent even after the task ends and the parent is closed. If an off-heap access exists in the parent iterator, it could cause segmentation fault which crashes the executor. Thus, we should use ContextAwareIterator to stop consuming after the task ends.

Since:
3.1.0

  • Nested Class Summary

    Nested classes/interfaces inherited from interface scala.collection.Iterator

    scala.collection.Iterator.GroupedIterator<B extends Object>, scala.collection.Iterator.SliceIterator<A extends Object>

  • Constructor Summary

    Constructors
    Constructor
    Description
    ContextAwareIterator(TaskContext context, scala.collection.Iterator<T> delegate)
    Deprecated.
     

  • Method Summary

    Modifier and Type
    Method
    Description
    TaskContext
    context()
    Deprecated.
     
    scala.collection.Iterator<T>
    delegate()
    Deprecated.
     
    boolean
    hasNext()
    Deprecated.
     
    T
    next()
    Deprecated.
     

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface scala.collection.IterableOnce

    knownSize, stepper

    Methods inherited from interface scala.collection.IterableOnceOps

    $colon$bslash, $div$colon, addString, addString, addString, aggregate, collectFirst, copyToArray, copyToArray, copyToArray, copyToBuffer, corresponds, count, exists, find, fold, foldLeft, foldRight, forall, foreach, isTraversableAgain, max, maxBy, maxByOption, maxOption, min, minBy, minByOption, minOption, mkString, mkString, mkString, nonEmpty, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, size, splitAt, sum, to, toArray, toBuffer, toIndexedSeq, toIterator, toList, toMap, toSeq, toSet, toStream, toVector

    Methods inherited from interface scala.collection.Iterator

    $plus$plus, buffered, collect, concat, contains, distinct, distinctBy, drop, dropWhile, duplicate, filter, filterImpl, filterNot, flatMap, flatten, grouped, hasDefiniteSize, indexOf, indexOf, indexWhere, indexWhere$default$2, isEmpty, iterator, length, map, nextOption, padTo, partition, patch, sameElements, scanLeft, scanRight, seq, slice, sliceIterator, sliding, sliding$default$2, span, take, takeWhile, tapEach, toString, withFilter, zip, zipAll, zipWithIndex

  • Constructor Details

    • ContextAwareIterator

      public ContextAwareIterator(TaskContext context, scala.collection.Iterator<T> delegate)
      Deprecated.

  • Method Details

    • context

      public TaskContext context()
      Deprecated.

    • delegate

      public scala.collection.Iterator<T> delegate()
      Deprecated.

    • hasNext

      public boolean hasNext()
      Deprecated.
      Specified by:
      hasNext in interface scala.collection.Iterator<T>

    • next

      public T next()
      Deprecated.
      Specified by:
      next in interface scala.collection.Iterator<T>