Type Members

1.  abstract class AccumulatorV2[IN, OUT] extends Serializable

   The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.

   The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.

   OUT should be a type that can be read atomically (e.g., Int, Long), or thread-safely (e.g., synchronized collections) because it will be read from other threads.

2.  class BestEffortLazyVal[T] extends Serializable

   A lock-free implementation of a lazily-initialized variable.

   A lock-free implementation of a lazily-initialized variable. If there are concurrent initializations then the compute() function may be invoked multiple times. However, only a single compute() result will be stored and all readers will receive the same result object instance.

   This may be helpful for avoiding deadlocks in certain scenarios where exactly-once value computation is not a hard requirement.

   Note

   This helper class has additional requirements on the compute function: 1) The compute function MUST not return null; 2) The computation failure is not cached.

   ,

   Scala 3 uses a different implementation of lazy vals which doesn't have this problem. Please refer to <a href="https://docs.scala-lang.org/scala3/reference/changed-features/lazy-vals-init.html">Lazy Vals Initialization for more details.

3.  class ChildFirstURLClassLoader extends MutableURLClassLoader

   A mutable class loader that gives preference to its own URLs over the parent class loader when loading classes and resources.

4.  class CollectionAccumulator[T] extends AccumulatorV2[T, List[T]]

   An accumulator for collecting a list of elements.

   An accumulator for collecting a list of elements.

   Since

   2.0.0

5.  class DoubleAccumulator extends AccumulatorV2[Double, Double]

   An accumulator for computing sum, count, and averages for double precision floating numbers.

   An accumulator for computing sum, count, and averages for double precision floating numbers.

   Since

   2.0.0

6.  class EnumUtil extends AnyRef

   Annotations

   @Private()

7.  final class ExposedBufferByteArrayOutputStream extends ByteArrayOutputStream

   Subclass of ByteArrayOutputStream that exposes buf directly.

8.  trait LastAttemptAccumulator[IN, OUT, PARTIAL] extends Logging

   A trait that can be mixed into a subclass of AccumulatorV2 to track the "logical" value of the "last attempt" of the execution using the accumulator - aggregated from the last attempts of any Task that calculated some RDD partitions and used this accumulator, and discarding any values coming from earlier attempts that have been recomputed.

   A trait that can be mixed into a subclass of AccumulatorV2 to track the "logical" value of the "last attempt" of the execution using the accumulator - aggregated from the last attempts of any Task that calculated some RDD partitions and used this accumulator, and discarding any values coming from earlier attempts that have been recomputed. If the accumulator is used by multiple RDDs, the last attempt value is tracked separately for each, and can be retrieved for each or all of them separately, see lastAttemptValueForX methods. If the accumulator is used directly on the Spark Driver using AccumulatorV2#add, that value is considered the last attempt value. If the accumulator was both used in Tasks and updated directly on the driver, it can't determine what should be considered the last attempt, and lastAttemptValueForX methods will return None.

   Contract for driver-only updates: A driver-side value (set via AccumulatorV2#add on the driver, outside any Task) is only returned by methods that do not narrow by RDD, namely lastAttemptValueForAllRDDs and lastAttemptValueForHighestRDDId. Methods that narrow to specific RDDs or RDD scopes (lastAttemptValueForRDDId, lastAttemptValueForRDDIds, lastAttemptValueForRDDScopes) return the zero value when a driver-only value is present, because a driver-side update cannot be attributed to any particular RDD or scope.

   LastAttemptAccumulator is not reset by the AccumulatorV2#reset method implementation, and its state is not copied by the AccumulatorV2#copy method implementation, and it should not be serialized to the Executors. The internal state should only be initialized by the initializeLastAttemptAccumulator method on the "main" instance of the accumulator, that was created and registered with AccumulatorContext with AccumulatorV2#register. All the interfaces of LastAttemptAccumulator: mergeLastAttempt (used only by DAGScheduler) and lastAttemptValueForX, logAccumulatorState (used by the using code) should only be invoked on that instance, on the Spark Driver.

   The LastAttemptAccumulator is not thread-safe. mergeLastAttempt should only be used by DAGScheduler, by the scheduler thread. Retrieving the value using lastAttemptValueForXXX while it is concurrently updated (execution is running) can produce some inconsistencies, but should not crash. If an RDD using the LastAttemptAccumulator is used concurrently by multiple actions that all try to recompute it, it may produce unexpected results and the semantics of what is "last attempt" becomes ambiguous. This should not be done in practice, and will likely result in more unexpected behaviours in Spark.

   Implementations must implement partialMergeVal and partialMerge methods operating on PARTIAL type. In regular AccumulatorV2 implementations, the AccumulatorV2 object itself holds the intermediate value of the accumulator, and AccumulatorV2#merge method is used to merge these objects together. LastAttemptAccumulator needs to keep track of partial values of every partition of every RDD that used the accumulator, and holding a full AccumulatorV2 object for each would have a high overhead. Therefore, an implementation should be able to return PARTIAL value from partialMergeVal that represents an intermediate mergeable value, and a partialMerge method that can merge that value into the accumulator. Implementations must also implement an isMergeable method that checks if the other AccumulatorV2 is of a compatible type to be merged with this using partialMergeVal. In regular AccumulatorV2 implementations, this check is normally done inside the AccumulatorV2#merge method, which is not used here.

   If an implementation is used to keep user data in the accumulator, it should override accumulatorStoresUserData to return true, to ensure correct structured logging annotation. Otherwise it should override it to false.

9.  trait LexicalThreadLocal[T] extends AnyRef

   Helper trait for defining thread locals with lexical scoping.

   Helper trait for defining thread locals with lexical scoping. With this helper, the thread local is private and can only be set by the Handle. The Handle only exposes the thread local value to functions passed into its runWith method. This pattern allows for the lifetime of the thread local value to be strictly controlled.

   Rather than calling tl.set(...) and tl.remove() you would get a handle and execute your code in handle.runWith { ... }.

   Example:

   object Credentials extends LexicalThreadLocal[Int] {
     def create(creds: Map[String, String]) = new Handle(Some(creds))
   }
   ...
   val handle = Credentials.create(Map("key" -> "value"))
   assert(Credentials.get() == None)
   handle.runWith {
     assert(Credentials.get() == Some(Map("key" -> "value")))
   }

10.  class LongAccumulator extends AccumulatorV2[Long, Long]

    An accumulator for computing sum, count, and average of 64-bit integers.

    An accumulator for computing sum, count, and average of 64-bit integers.

    Since

    2.0.0

11.  case class MutablePair[T1, T2](_1: T1, _2: T2) extends Product2[T1, T2] with Product with Serializable

    :: DeveloperApi :: A tuple of 2 elements.

    :: DeveloperApi :: A tuple of 2 elements. This can be used as an alternative to Scala's Tuple2 when we want to minimize object allocation.

    _1

    Element 1 of this MutablePair

    _2

    Element 2 of this MutablePair

    Annotations

    @DeveloperApi()

12.  class MutableURLClassLoader extends URLClassLoader

    URL class loader that exposes the addURL method in URLClassLoader.

13.  final class Pair[L, R] extends Record

    An immutable pair of values.

    An immutable pair of values. Note that the fields are intentionally designed to be getLeft and getRight instead of left and right in order to mitigate the migration burden from org.apache.commons.lang3.tuple.Pair.

14.  class ParentClassLoader extends ClassLoader

    A class loader which makes some protected methods in ClassLoader accessible.

15.  class SerializableConfiguration extends Serializable

    Hadoop configuration but serializable.

    Hadoop configuration but serializable. Use value to access the Hadoop configuration.

    Annotations

    @DeveloperApi() @Unstable()

16.  class StatCounter extends Serializable

    A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way.

    A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way. Includes support for merging two StatCounters. Based on Welford and Chan's algorithms for running variance.

17.  trait TaskCompletionListener extends EventListener

    :: DeveloperApi ::

    :: DeveloperApi ::

    Listener providing a callback function to invoke when a task's execution completes.

    Annotations

    @DeveloperApi()

18.  trait TaskFailureListener extends EventListener

    :: DeveloperApi ::

    :: DeveloperApi ::

    Listener providing a callback function to invoke when a task's execution encounters an error. Operations defined here must be idempotent, as onTaskFailure can be called multiple times.

    Annotations

    @DeveloperApi()

19.  trait TaskInterruptListener extends EventListener

    :: DeveloperApi ::

    :: DeveloperApi ::

    Listener providing a callback function to invoke when a task's execution is interrupted.

    Annotations

    @DeveloperApi()

20.  class TransientBestEffortLazyVal[T] extends Serializable

    A lock-free implementation of a lazily-initialized variable.

    A lock-free implementation of a lazily-initialized variable. If there are concurrent initializations then the compute() function may be invoked multiple times. However, only a single compute() result will be stored and all readers will receive the same result object instance.

    This may be helpful for avoiding deadlocks in certain scenarios where exactly-once value computation is not a hard requirement.

    The main difference between this and BestEffortLazyVal is that: BestEffortLazyVal serializes the cached value after computation, while TransientBestEffortLazyVal always serializes the compute function.

    Note

    This helper class has additional requirements on the compute function: 1) The compute function MUST not return null; 2) The computation failure is not cached.

    ,

    Scala 3 uses a different implementation of lazy vals which doesn't have this problem. Please refer to <a href="https://docs.scala-lang.org/scala3/reference/changed-features/lazy-vals-init.html">Lazy Vals Initialization for more details.

21.  final class UUIDv7Generator extends AnyRef

    Generator for UUIDv7 as defined in RFC 9562.

    Generator for UUIDv7 as defined in RFC 9562. https://www.rfc-editor.org/rfc/rfc9562.html

    UUIDv7 is a time-ordered UUID that embeds a Unix timestamp in milliseconds.

    Monotonicity is best-effort but not strictly guaranteed by this implementation, in rare cases such as concurrent generation within the same millisecond or clock adjustments. This trade-off is intentional to avoid throughput degradation or thread contention.