:: DeveloperApi :: A set of functions used to aggregate data.

:: Experimental :: A TaskContext with extra contextual info and tooling for tasks in a barrier stage. Use BarrierTaskContext#get to obtain the barrier context for a running barrier task.

:: Experimental :: Carries all task infos of a barrier task.

A FutureAction for actions that could trigger multiple Spark jobs. Examples include take, takeSample. Cancellation works by setting the cancelled flag to true and cancelling any pending jobs.

:: DeveloperApi :: Base class for dependencies.

A reader to load error information from one or more JSON files. Note that, if one error appears in more than one JSON files, the latter wins. Please read common/utils/src/main/resources/error/README.md for more details.

:: DeveloperApi :: Task failed due to a runtime exception. This is the most common failure case and also captures user program exceptions.

stackTrace contains the stack trace of the exception itself. It still exists for backward compatibility. It's better to use this(e: Throwable, metrics: Option[TaskMetrics]) to create ExceptionFailure as it will handle the backward compatibility properly.

fullStackTrace is a better representation of the stack trace because it contains the whole stack trace including the exception and its causes

exception is the actual exception that caused the task to fail. It may be None in the case that the exception is not in fact serializable. If a task fails more than once (due to retries), exception is that one that caused the last failure.

:: DeveloperApi :: The task failed because the executor that it was running on was lost. This may happen because the task crashed the JVM.

:: DeveloperApi :: Task failed to fetch shuffle data from a remote node. Probably means we have lost the remote executors the task is trying to fetch from, and thus need to rerun the previous stage.

A future for the result of an action to support cancellation. This is an extension of the Scala Future interface to support cancellation.

A org.apache.spark.Partitioner that implements hash-based partitioning using Java's Object.hashCode.

Java arrays have hashCodes that are based on the arrays' identities rather than their contents, so attempting to partition an RDD[Array[_]] or RDD[(Array[_], _)] using a HashPartitioner will produce an unexpected or incorrect result.

:: DeveloperApi :: An iterator that wraps around an existing iterator to provide task killing functionality. It works by checking the interrupted flag in TaskContext.

Handle via which a "run" function passed to a ComplexFutureAction can submit jobs for execution.

:: DeveloperApi :: Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD. Narrow dependencies allow for pipelined execution.

:: DeveloperApi :: Represents a one-to-one dependency between partitions of the parent and child RDDs.

An evaluator for computing RDD partitions. Spark serializes and sends PartitionEvaluatorFactory to executors, and then creates PartitionEvaluator via the factory at the executor side.

A factory to create PartitionEvaluator. Spark serializes and sends PartitionEvaluatorFactory to executors, and then creates PartitionEvaluator via the factory at the executor side.

An object that defines how the elements in a key-value pair RDD are partitioned by key. Maps each key to a partition ID, from 0 to numPartitions - 1.

Note that, partitioner must be deterministic, i.e. it must return the same partition id given the same partition key.

Query context of a SparkThrowable. It helps users understand where error occur while executing queries.

The type of QueryContext.

:: DeveloperApi :: Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.

A org.apache.spark.Partitioner that partitions sortable records by range into roughly equal ranges. The ranges are determined by sampling the content of the RDD passed in.

:: DeveloperApi :: Represents a dependency on the output of a shuffle stage. Note that in the case of shuffle, the RDD is transient since we don't need it on the executor side.

A FutureAction holding the result of an action that triggers a single job. Examples include count, collect, reduce.

Configuration for a Spark application. Used to set various Spark parameters as key-value pairs.

Most of the time, you would create a SparkConf object with new SparkConf(), which will load values from any spark.* Java system properties set in your application as well. In this case, parameters you set directly on the SparkConf object take priority over system properties.

For unit tests, you can also call new SparkConf(false) to skip loading external settings and get the same configuration no matter what the system properties are.

All setter methods in this class support chaining. For example, you can write new SparkConf().setMaster("local").setAppName("My app").

Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster, and can be used to create RDDs, accumulators and broadcast variables on that cluster.

:: DeveloperApi :: Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc. Currently Spark code finds the SparkEnv through a global variable, so all the threads can access the same SparkEnv. It can be accessed by SparkEnv.get (e.g. after creating a SparkContext).

Exposes information about Spark Executors.

This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.

Class that allows users to receive all SparkListener events. Users should override the onEvent method.

This is a concrete Java class in order to ensure that we don't forget to update it when adding new methods to SparkListener: forgetting to add a method will result in a compilation error (if this was a concrete Scala class, default implementations of new event handlers would be inherited from the SparkListener trait).

Please note until Spark 3.1.0 this was missing the DevelopApi annotation, this needs to be taken into account if changing this API before a major release.

Exposes information about Spark Jobs.

This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.

Exposes information about Spark Stages.

This interface is not designed to be implemented outside of Spark. We may add additional methods which may break binary compatibility with outside implementations.

Low-level status reporting APIs for monitoring job and stage progress.

These APIs intentionally provide very weak consistency semantics; consumers of these APIs should be prepared to handle empty / missing information. For example, a job's stage ids may be known but the status API may not have any information about the details of those stages, so getStageInfo could potentially return None for a valid stage id.

To limit memory usage, these APIs only provide information on recent jobs / stages. These APIs will provide information for the last spark.ui.retainedStages stages and spark.ui.retainedJobs jobs.

NOTE: this class's constructor should be considered private and may be subject to change.

Interface mixed into Throwables thrown from Spark.

- For backwards compatibility, existing Throwable types can be thrown with an arbitrary error message with a null error class. See SparkException. - To promote standardization, Throwables should be thrown with an error class and message parameters to construct an error message with SparkThrowableHelper.getMessage(). New Throwable types should not accept arbitrary error messages. See SparkArithmeticException.

:: DeveloperApi :: Task requested the driver to commit, but was denied.

Contextual information about a task which can be read or mutated during execution. To access the TaskContext for a running task, use:

org.apache.spark.TaskContext.get()

:: DeveloperApi :: Various possible reasons why a task ended. The low-level TaskScheduler is supposed to retry tasks several times for "ephemeral" failures, and only report back failures that require some old stages to be resubmitted, such as shuffle map fetch failures.

:: DeveloperApi :: Various possible reasons why a task failed.

:: DeveloperApi :: Task was killed intentionally and needs to be rescheduled.

:: DeveloperApi :: Exception thrown when a task is explicitly killed (i.e., task failure is expected).

:: DeveloperApi :: A TaskContext aware iterator.

As the Python evaluation consumes the parent iterator in a separate thread, it could consume more data from the parent even after the task ends and the parent is closed. If an off-heap access exists in the parent iterator, it could cause segmentation fault which crashes the executor. Thus, we should use ContextAwareIterator to stop consuming after the task ends.