org.apache.spark

Class ContextAwareIterator<T>

Object

org.apache.spark.ContextAwareIterator<T>

All Implemented Interfaces:

scala.collection.GenTraversableOnce<T>, scala.collection.Iterator<T>, scala.collection.TraversableOnce<T>

public class ContextAwareIterator<T>
extends Object
implements scala.collection.Iterator<T>

:: DeveloperApi :: A TaskContext aware iterator.

As the Python evaluation consumes the parent iterator in a separate thread, it could consume more data from the parent even after the task ends and the parent is closed. If an off-heap access exists in the parent iterator, it could cause segmentation fault which crashes the executor. Thus, we should use ContextAwareIterator to stop consuming after the task ends.

Since:

3.1.0

Nested Class Summary

Nested classes/interfaces inherited from interface scala.collection.Iterator

scala.collection.Iterator.GroupedIterator<B>, scala.collection.Iterator.SliceIterator<A>

Nested classes/interfaces inherited from interface scala.collection.TraversableOnce

scala.collection.TraversableOnce.BufferedCanBuildFrom<A,CC extends scala.collection.TraversableOnce<Object>>, scala.collection.TraversableOnce.FlattenOps<A>, scala.collection.TraversableOnce.ForceImplicitAmbiguity, scala.collection.TraversableOnce.MonadOps<A>, scala.collection.TraversableOnce.OnceCanBuildFrom<A>

Constructor Summary

Constructors

Constructor and Description
ContextAwareIterator(TaskContext context, scala.collection.Iterator<T> delegate)

Method Summary

Modifier and Type	Method and Description
TaskContext	context()
scala.collection.Iterator<T>	delegate()
boolean	hasNext()
T	next()

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface scala.collection.Iterator

$init$, $plus$plus, apply, buffered, collect, contains, continually, copyToArray, corresponds, drop, dropWhile, duplicate, empty, exists, fill, filter, filterNot, find, flatMap, forall, foreach, from, from, grouped, hasDefiniteSize, indexOf, indexOf, indexWhere, indexWhere, isEmpty, isTraversableAgain, iterate, IteratorCanBuildFrom, length, map, padTo, partition, patch, range, range, sameElements, scanLeft, scanRight, seq, single, slice, sliceIterator, sliding, sliding$default$2, span, tabulate, take, takeWhile, toIterator, toStream, toString, toTraversable, withFilter, zip, zipAll, zipWithIndex

Methods inherited from interface scala.collection.TraversableOnce

$colon$bslash, $div$colon, $init$, addString, addString, addString, aggregate, alternateImplicit, collectFirst, copyToArray, copyToArray, copyToBuffer, count, flattenTraversableOnce, fold, foldLeft, foldRight, max, maxBy, min, minBy, mkString, mkString, mkString, MonadOps, nonEmpty, OnceCanBuildFrom, product, reduce, reduceLeft, reduceLeftOption, reduceOption, reduceRight, reduceRightOption, reversed, size, sum, to, toArray, toBuffer, toIndexedSeq, toIterable, toList, toMap, toSeq, toSet, toVector

Methods inherited from interface scala.collection.GenTraversableOnce

$init$, sizeHintIfCheap

Constructor Detail

ContextAwareIterator

public ContextAwareIterator(TaskContext context,
                            scala.collection.Iterator<T> delegate)

Method Detail

context

public TaskContext context()

delegate

public scala.collection.Iterator<T> delegate()

hasNext

public boolean hasNext()

Specified by:

hasNext in interface scala.collection.Iterator<T>

next

public T next()

Specified by:

next in interface scala.collection.Iterator<T>