A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ 

A

abort(String) - Method in class org.apache.spark.scheduler.TaskSetManager
 
abortStage(Stage, String) - Method in class org.apache.spark.scheduler.DAGScheduler
Aborts all jobs depending on a particular Stage.
AbsoluteError - Class in org.apache.spark.mllib.tree.loss
:: DeveloperApi :: Class for absolute error loss calculation (for regression).
AbsoluteError() - Constructor for class org.apache.spark.mllib.tree.loss.AbsoluteError
 
accept(File, String) - Method in class org.apache.spark.rdd.PipedRDD.NotEqualsFileNameFilter
 
AcceptanceResult - Class in org.apache.spark.util.random
Object used by seqOp to keep track of the number of items accepted and items waitlisted per stratum, as well as the bounds for accepting and waitlisting items.
AcceptanceResult(long, long) - Constructor for class org.apache.spark.util.random.AcceptanceResult
 
acceptBound() - Method in class org.apache.spark.util.random.AcceptanceResult
 
Accumulable<R,T> - Class in org.apache.spark
A data type that can be accumulated, ie has an commutative and associative "add" operation, but where the result type, R, may be different from the element type being added, T.
Accumulable(R, AccumulableParam<R, T>, Option<String>) - Constructor for class org.apache.spark.Accumulable
 
Accumulable(R, AccumulableParam<R, T>) - Constructor for class org.apache.spark.Accumulable
 
accumulable(T, AccumulableParam<T, R>) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulable shared variable of the given type, to which tasks can "add" values with add.
accumulable(T, String, AccumulableParam<T, R>) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulable shared variable of the given type, to which tasks can "add" values with add.
accumulable(R, AccumulableParam<R, T>) - Method in class org.apache.spark.SparkContext
Create an Accumulable shared variable, to which tasks can add values with +=.
accumulable(R, String, AccumulableParam<R, T>) - Method in class org.apache.spark.SparkContext
Create an Accumulable shared variable, with a name for display in the Spark UI.
accumulableCollection(R, Function1<R, Growable<T>>, ClassTag<R>) - Method in class org.apache.spark.SparkContext
Create an accumulator from a "mutable collection" type.
AccumulableInfo - Class in org.apache.spark.scheduler
:: DeveloperApi :: Information about an Accumulable modified during a task or stage.
AccumulableInfo(long, String, Option<String>, String) - Constructor for class org.apache.spark.scheduler.AccumulableInfo
 
accumulableInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
accumulableInfoToJson(AccumulableInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
AccumulableParam<R,T> - Interface in org.apache.spark
Helper object defining how to accumulate values of a particular type.
accumulables() - Method in class org.apache.spark.scheduler.StageInfo
Terminal values of accumulables updated during this stage.
accumulables() - Method in class org.apache.spark.scheduler.TaskInfo
Intermediate updates to accumulables during this task.
accumulables() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
Accumulator<T> - Class in org.apache.spark
A simpler value of Accumulable where the result type being accumulated is the same as the types of elements being merged, i.e.
Accumulator(T, AccumulatorParam<T>, Option<String>) - Constructor for class org.apache.spark.Accumulator
 
Accumulator(T, AccumulatorParam<T>) - Constructor for class org.apache.spark.Accumulator
 
accumulator(int) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator integer variable, which tasks can "add" values to using the add method.
accumulator(int, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator integer variable, which tasks can "add" values to using the add method.
accumulator(double) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator double variable, which tasks can "add" values to using the add method.
accumulator(double, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator double variable, which tasks can "add" values to using the add method.
accumulator(T, AccumulatorParam<T>) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator variable of a given type, which tasks can "add" values to using the add method.
accumulator(T, String, AccumulatorParam<T>) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator variable of a given type, which tasks can "add" values to using the add method.
accumulator(T, AccumulatorParam<T>) - Method in class org.apache.spark.SparkContext
Create an Accumulator variable of a given type, which tasks can "add" values to using the += method.
accumulator(T, String, AccumulatorParam<T>) - Method in class org.apache.spark.SparkContext
Create an Accumulator variable of a given type, with a name for display in the Spark UI.
accumulator() - Method in class org.apache.spark.sql.execution.PythonUDF
 
AccumulatorParam<T> - Interface in org.apache.spark
A simpler version of AccumulableParam where the only data type you can add in is the same type as the accumulated value.
Accumulators - Class in org.apache.spark
 
Accumulators() - Constructor for class org.apache.spark.Accumulators
 
accumUpdates() - Method in class org.apache.spark.scheduler.CompletionEvent
 
accumUpdates() - Method in class org.apache.spark.scheduler.DirectTaskResult
 
accuracy() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns accuracy
aclsEnabled() - Method in class org.apache.spark.SecurityManager
Check to see if Acls for the UI are enabled
active() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
activeExecutorIds() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
ActiveJob - Class in org.apache.spark.scheduler
Tracks information about an active job in the DAGScheduler.
ActiveJob(int, Stage, Function2<TaskContext, Iterator<Object>, ?>, int[], CallSite, JobListener, Properties) - Constructor for class org.apache.spark.scheduler.ActiveJob
 
activeJobs() - Method in class org.apache.spark.scheduler.DAGScheduler
 
activeJobs() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
activeStages() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
activeTasks() - Method in class org.apache.spark.ui.exec.ExecutorSummaryInfo
 
activeTaskSets() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
actor() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
ACTOR_NAME() - Static method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
ACTOR_NAME() - Static method in class org.apache.spark.scheduler.cluster.YarnSchedulerBackend
 
ActorHelper - Interface in org.apache.spark.streaming.receiver
:: DeveloperApi :: A receiver trait to be mixed in with your Actor to gain access to the API for pushing received data into Spark Streaming for being processed.
ActorLogReceive - Interface in org.apache.spark.util
A trait to enable logging all Akka actor messages.
ActorReceiver<T> - Class in org.apache.spark.streaming.receiver
Provides Actors as receivers for receiving stream.
ActorReceiver(Props, String, StorageLevel, SupervisorStrategy, ClassTag<T>) - Constructor for class org.apache.spark.streaming.receiver.ActorReceiver
 
ActorReceiver.Supervisor - Class in org.apache.spark.streaming.receiver
 
ActorReceiver.Supervisor() - Constructor for class org.apache.spark.streaming.receiver.ActorReceiver.Supervisor
 
ActorReceiverData - Interface in org.apache.spark.streaming.receiver
Case class to receive data sent by child actors
actorStream(Props, String, StorageLevel, SupervisorStrategy) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream with any arbitrary user implemented actor receiver.
actorStream(Props, String, StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream with any arbitrary user implemented actor receiver.
actorStream(Props, String) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream with any arbitrary user implemented actor receiver.
actorStream(Props, String, StorageLevel, SupervisorStrategy, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream with any arbitrary user implemented actor receiver.
ActorSupervisorStrategy - Class in org.apache.spark.streaming.receiver
:: DeveloperApi :: A helper with set of defaults for supervisor strategy
ActorSupervisorStrategy() - Constructor for class org.apache.spark.streaming.receiver.ActorSupervisorStrategy
 
actorSystem() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
actorSystem() - Method in class org.apache.spark.SparkEnv
 
actualSize(Row, int) - Method in class org.apache.spark.sql.columnar.ByteArrayColumnType
 
actualSize(Row, int) - Method in class org.apache.spark.sql.columnar.ColumnType
Returns the size of the value row(ordinal).
actualSize(Row, int) - Static method in class org.apache.spark.sql.columnar.STRING
 
add(T) - Method in class org.apache.spark.Accumulable
Add more data to this accumulator / accumulable
add(Map<Object, Object>) - Static method in class org.apache.spark.Accumulators
 
add(long, long, ED) - Method in class org.apache.spark.graphx.impl.EdgePartitionBuilder
Add a new edge to the partition.
add(long, long, int, int, ED) - Method in class org.apache.spark.graphx.impl.ExistingEdgePartitionBuilder
Add a new edge to the partition.
add(Vector) - Method in class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
Adds a new document.
add(Vector) - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Add a new sample to this summarizer, and update the statistical summary.
add(ImpurityCalculator) - Method in class org.apache.spark.mllib.tree.impurity.ImpurityCalculator
Add the stats from another calculator into this one, modifying and returning this calculator.
add(long, long) - Static method in class org.apache.spark.streaming.util.RawTextHelper
 
add(Vector) - Method in class org.apache.spark.util.Vector
 
addAccumulator(R, T) - Method in interface org.apache.spark.AccumulableParam
Add additional data to the accumulator value.
addAccumulator(T, T) - Method in interface org.apache.spark.AccumulatorParam
 
addAccumulator(R, T) - Method in class org.apache.spark.GrowableAccumulableParam
 
addBinary(Binary) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addBlock(BlockId, BlockStatus) - Method in class org.apache.spark.storage.StorageStatus
Add the given block to this storage status.
AddBlock - Class in org.apache.spark.streaming.scheduler
 
AddBlock(ReceivedBlockInfo) - Constructor for class org.apache.spark.streaming.scheduler.AddBlock
 
addBlock(ReceivedBlockInfo) - Method in class org.apache.spark.streaming.scheduler.ReceivedBlockTracker
Add received block.
addBoolean(boolean) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addData(Object) - Method in class org.apache.spark.streaming.receiver.BlockGenerator
Push a single data item into the buffer.
addDataWithCallback(Object, Object) - Method in class org.apache.spark.streaming.receiver.BlockGenerator
Push a single data item into the buffer.
addDouble(double) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addedFiles() - Method in class org.apache.spark.SparkContext
 
addedJars() - Method in class org.apache.spark.SparkContext
 
AddExchange - Class in org.apache.spark.sql.execution
Ensures that the Partitioning of input data meets the Distribution requirements for each operator by inserting Exchange Operators where required.
AddExchange(SQLContext) - Constructor for class org.apache.spark.sql.execution.AddExchange
 
addFile(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Add a file to be downloaded with this Spark job on every node.
addFile(File) - Method in class org.apache.spark.HttpFileServer
 
addFile(String) - Method in class org.apache.spark.SparkContext
Add a file to be downloaded with this Spark job on every node.
AddFile - Class in org.apache.spark.sql.hive
 
AddFile(String) - Constructor for class org.apache.spark.sql.hive.AddFile
 
AddFile - Class in org.apache.spark.sql.hive.execution
:: DeveloperApi ::
AddFile(String) - Constructor for class org.apache.spark.sql.hive.execution.AddFile
 
addFileToDir(File, File) - Method in class org.apache.spark.HttpFileServer
 
addFilters(Seq<ServletContextHandler>, SparkConf) - Static method in class org.apache.spark.ui.JettyUtils
Add filters, if any, to the given list of ServletContextHandlers
addFloat(float) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addGrid(Param<T>, Iterable<T>) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a param with multiple values (overwrites if the input param exists).
addGrid(DoubleParam, double[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a double param with multiple values.
addGrid(IntParam, int[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a int param with multiple values.
addGrid(FloatParam, float[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a float param with multiple values.
addGrid(LongParam, long[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a long param with multiple values.
addGrid(BooleanParam) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a boolean param with true and false.
addInPlace(R, R) - Method in interface org.apache.spark.AccumulableParam
Merge two accumulated values together.
addInPlace(R, R) - Method in class org.apache.spark.GrowableAccumulableParam
 
addInPlace(double, double) - Method in class org.apache.spark.SparkContext.DoubleAccumulatorParam$
 
addInPlace(float, float) - Method in class org.apache.spark.SparkContext.FloatAccumulatorParam$
 
addInPlace(int, int) - Method in class org.apache.spark.SparkContext.IntAccumulatorParam$
 
addInPlace(long, long) - Method in class org.apache.spark.SparkContext.LongAccumulatorParam$
 
addInPlace(Vector) - Method in class org.apache.spark.util.Vector
 
addInPlace(Vector, Vector) - Method in class org.apache.spark.util.Vector.VectorAccumParam$
 
addInputStream(InputDStream<?>) - Method in class org.apache.spark.streaming.DStreamGraph
 
addInt(int) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addJar(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
addJar(File) - Method in class org.apache.spark.HttpFileServer
 
addJar(String) - Method in class org.apache.spark.SparkContext
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
AddJar - Class in org.apache.spark.sql.hive
 
AddJar(String) - Constructor for class org.apache.spark.sql.hive.AddJar
 
AddJar - Class in org.apache.spark.sql.hive.execution
:: DeveloperApi ::
AddJar(String) - Constructor for class org.apache.spark.sql.hive.execution.AddJar
 
addListener(SparkListener) - Method in interface org.apache.spark.scheduler.SparkListenerBus
 
addListener(StreamingListener) - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBus
 
addLocalConfiguration(String, int, int, int, JobConf) - Static method in class org.apache.spark.rdd.HadoopRDD
Add Hadoop configuration specific to a single partition and attempt.
addLong(long) - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
addOnCompleteCallback(Function0<Unit>) - Method in class org.apache.spark.TaskContext
Deprecated.
addOnCompleteCallback(Function0<BoxedUnit>) - Method in class org.apache.spark.TaskContextImpl
 
addOutputLoc(int, MapStatus) - Method in class org.apache.spark.scheduler.Stage
 
addOutputStream(DStream<?>) - Method in class org.apache.spark.streaming.DStreamGraph
 
addPartitioningAttributes(Seq<Attribute>) - Method in class org.apache.spark.sql.hive.HiveStrategies.ParquetConversion.LogicalPlanHacks
 
addPartToPGroup(Partition, PartitionGroup) - Method in class org.apache.spark.rdd.PartitionCoalescer
 
address() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.FetchRequest
 
addresses() - Method in class org.apache.spark.streaming.flume.FlumePollingInputDStream
 
addRunningTask(long) - Method in class org.apache.spark.scheduler.TaskSetManager
If the given task ID is not in the set of running tasks, adds it.
addSchedulable(Schedulable) - Method in class org.apache.spark.scheduler.Pool
 
addSchedulable(Schedulable) - Method in interface org.apache.spark.scheduler.Schedulable
 
addSchedulable(Schedulable) - Method in class org.apache.spark.scheduler.TaskSetManager
 
addSparkListener(SparkListener) - Method in class org.apache.spark.SparkContext
:: DeveloperApi :: Register a listener to receive up-calls from events that happen during execution.
addStreamingListener(StreamingListener) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Add a StreamingListener object for receiving system events related to streaming.
addStreamingListener(StreamingListener) - Method in class org.apache.spark.streaming.StreamingContext
Add a StreamingListener object for receiving system events related to streaming.
addTaskCompletionListener(TaskCompletionListener) - Method in class org.apache.spark.TaskContext
Add a (Java friendly) listener to be executed on task completion.
addTaskCompletionListener(Function1<TaskContext, Unit>) - Method in class org.apache.spark.TaskContext
Add a listener in the form of a Scala closure to be executed on task completion.
addTaskCompletionListener(TaskCompletionListener) - Method in class org.apache.spark.TaskContextImpl
 
addTaskCompletionListener(Function1<TaskContext, BoxedUnit>) - Method in class org.apache.spark.TaskContextImpl
 
addTaskSetManager(Schedulable, Properties) - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
addTaskSetManager(Schedulable, Properties) - Method in class org.apache.spark.scheduler.FIFOSchedulableBuilder
 
addTaskSetManager(Schedulable, Properties) - Method in interface org.apache.spark.scheduler.SchedulableBuilder
 
addToTime(long) - Method in class org.apache.spark.streaming.util.ManualClock
 
adminAcls() - Method in class org.apache.spark.scheduler.ApplicationEventListener
 
advanceCheckpoint() - Method in class org.apache.spark.streaming.kinesis.KinesisCheckpointState
Advance the checkpoint clock by the checkpoint interval.
aggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".
Aggregate - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Groups input data by groupingExpressions and computes the aggregateExpressions for each group.
Aggregate(boolean, Seq<Expression>, Seq<NamedExpression>, SparkPlan) - Constructor for class org.apache.spark.sql.execution.Aggregate
 
aggregate() - Method in class org.apache.spark.sql.execution.Aggregate.ComputedAggregate
 
aggregate(Seq<Expression>) - Method in class org.apache.spark.sql.SchemaRDD
Performs an aggregation over all Rows in this RDD.
Aggregate.ComputedAggregate - Class in org.apache.spark.sql.execution
An aggregate that needs to be computed for each row in a group.
Aggregate.ComputedAggregate(AggregateExpression, AggregateExpression, AttributeReference) - Constructor for class org.apache.spark.sql.execution.Aggregate.ComputedAggregate
 
Aggregate.ComputedAggregate$ - Class in org.apache.spark.sql.execution
 
Aggregate.ComputedAggregate$() - Constructor for class org.apache.spark.sql.execution.Aggregate.ComputedAggregate$
 
aggregateByKey(U, Partitioner, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, int, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Partitioner, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, int, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
AggregateEvaluation - Class in org.apache.spark.sql.execution
 
AggregateEvaluation(Seq<Attribute>, Seq<Expression>, Seq<Expression>, Expression) - Constructor for class org.apache.spark.sql.execution.AggregateEvaluation
 
aggregateExpressions() - Method in class org.apache.spark.sql.execution.Aggregate
 
aggregateExpressions() - Method in class org.apache.spark.sql.execution.GeneratedAggregate
 
aggregateMessages(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, ClassTag<A>) - Method in class org.apache.spark.graphx.Graph
Aggregates values from the neighboring edges and vertices of each vertex.
aggregateMessagesEdgeScan(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, EdgeActiveness, ClassTag<A>) - Method in class org.apache.spark.graphx.impl.EdgePartition
Send messages along edges and aggregate them at the receiving vertices.
aggregateMessagesIndexScan(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, EdgeActiveness, ClassTag<A>) - Method in class org.apache.spark.graphx.impl.EdgePartition
Send messages along edges and aggregate them at the receiving vertices.
aggregateMessagesWithActiveSet(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, Option<Tuple2<VertexRDD<?>, EdgeDirection>>, ClassTag<A>) - Method in class org.apache.spark.graphx.Graph
Aggregates values from the neighboring edges and vertices of each vertex.
aggregateMessagesWithActiveSet(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, Option<Tuple2<VertexRDD<?>, EdgeDirection>>, ClassTag<A>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
aggregateSizeForNode(DecisionTreeMetadata, Option<int[]>) - Static method in class org.apache.spark.mllib.tree.RandomForest
Get the number of values to be stored for this node in the bin aggregates.
aggregateUsingIndex(Iterator<Product2<Object, VD2>>, Function2<VD2, VD2, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexPartitionBaseOps
 
aggregateUsingIndex(RDD<Tuple2<Object, VD2>>, Function2<VD2, VD2, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
aggregateUsingIndex(RDD<Tuple2<Object, VD2>>, Function2<VD2, VD2, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Aggregates vertices in messages that have the same ids using reduceFunc, returning a VertexRDD co-indexed with this.
AggregatingEdgeContext<VD,ED,A> - Class in org.apache.spark.graphx.impl
 
AggregatingEdgeContext(Function2<A, A, A>, Object, BitSet) - Constructor for class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
Aggregator<K,V,C> - Class in org.apache.spark
:: DeveloperApi :: A set of functions used to aggregate data.
Aggregator(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Constructor for class org.apache.spark.Aggregator
 
aggregator() - Method in class org.apache.spark.ShuffleDependency
 
AkkaUtils - Class in org.apache.spark.util
Various utility classes for working with Akka.
AkkaUtils() - Constructor for class org.apache.spark.util.AkkaUtils
 
Algo - Class in org.apache.spark.mllib.tree.configuration
:: Experimental :: Enum to select the algorithm for the decision tree
Algo() - Constructor for class org.apache.spark.mllib.tree.configuration.Algo
 
algo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
algo() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
algo() - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
algo() - Method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
algorithm() - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
 
alias() - Method in class org.apache.spark.sql.hive.MetastoreRelation
 
aliasNames() - Method in class org.apache.spark.sql.hive.HiveGenericUdtf
 
All - Static variable in class org.apache.spark.graphx.TripletFields
Expose all the fields (source, edge, and destination).
allAggregates(Seq<Expression>) - Method in class org.apache.spark.sql.execution.SparkStrategies.HashAggregation
 
AllCompressionSchemes - Interface in org.apache.spark.sql.columnar.compression
 
AllJobsCancelled - Class in org.apache.spark.scheduler
 
AllJobsCancelled() - Constructor for class org.apache.spark.scheduler.AllJobsCancelled
 
AllJobsPage - Class in org.apache.spark.ui.jobs
Page showing list of all ongoing and recently finished jobs
AllJobsPage(JobsTab) - Constructor for class org.apache.spark.ui.jobs.AllJobsPage
 
allJoinTokens() - Static method in class org.apache.spark.sql.hive.HiveQl
 
allocateBlocksToBatch(Time) - Method in class org.apache.spark.streaming.scheduler.ReceivedBlockTracker
Allocate all unallocated blocks to the given batch.
allocateBlocksToBatch(Time) - Method in class org.apache.spark.streaming.scheduler.ReceiverTracker
Allocate all unallocated blocks to the given batch.
AllocatedBlocks - Class in org.apache.spark.streaming.scheduler
Class representing the blocks of all the streams allocated to a batch
AllocatedBlocks(Map<Object, Seq<ReceivedBlockInfo>>) - Constructor for class org.apache.spark.streaming.scheduler.AllocatedBlocks
 
allocatedBlocks() - Method in class org.apache.spark.streaming.scheduler.BatchAllocationEvent
 
allowExisting() - Method in class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
allowLocal() - Method in class org.apache.spark.scheduler.JobSubmitted
 
allPendingTasks() - Method in class org.apache.spark.scheduler.TaskSetManager
 
AllStagesPage - Class in org.apache.spark.ui.jobs
Page showing list of all ongoing and recently finished stages and pools
AllStagesPage(StagesTab) - Constructor for class org.apache.spark.ui.jobs.AllStagesPage
 
AlphaComponent - Annotation Type in org.apache.spark.annotation
A new component of Spark which may have unstable API's.
alreadyPlanned() - Method in class org.apache.spark.sql.execution.SparkLogicalPlan
 
ALS - Class in org.apache.spark.mllib.recommendation
Alternating Least Squares matrix factorization.
ALS() - Constructor for class org.apache.spark.mllib.recommendation.ALS
Constructs an ALS instance with default parameters: {numBlocks: -1, rank: 10, iterations: 10, lambda: 0.01, implicitPrefs: false, alpha: 1.0}.
ALS.BlockStats - Class in org.apache.spark.mllib.recommendation
:: DeveloperApi :: Statistics of a block in ALS computation.
ALS.BlockStats(String, int, long, long, long, long) - Constructor for class org.apache.spark.mllib.recommendation.ALS.BlockStats
 
ALS.BlockStats$ - Class in org.apache.spark.mllib.recommendation
 
ALS.BlockStats$() - Constructor for class org.apache.spark.mllib.recommendation.ALS.BlockStats$
 
ALSPartitioner - Class in org.apache.spark.mllib.recommendation
Partitioner for ALS.
ALSPartitioner(int) - Constructor for class org.apache.spark.mllib.recommendation.ALSPartitioner
 
analyze(String) - Method in class org.apache.spark.sql.hive.HiveContext
Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.
analyzeBlocks(RDD<Rating>, int, int) - Static method in class org.apache.spark.mllib.recommendation.ALS
:: DeveloperApi :: Given an RDD of ratings, number of user blocks, and number of product blocks, computes the statistics of each block in ALS computation.
analyzed() - Method in class org.apache.spark.sql.hive.test.TestHiveContext.QueryExecution
 
AnalyzeTable - Class in org.apache.spark.sql.hive
 
AnalyzeTable(String) - Constructor for class org.apache.spark.sql.hive.AnalyzeTable
 
AnalyzeTable - Class in org.apache.spark.sql.hive.execution
:: DeveloperApi :: Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.
AnalyzeTable(String) - Constructor for class org.apache.spark.sql.hive.execution.AnalyzeTable
 
AND() - Static method in class org.apache.spark.sql.hive.HiveQl
 
ANY() - Static method in class org.apache.spark.scheduler.TaskLocality
 
append(boolean, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BOOLEAN
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BOOLEAN
 
append(byte, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BYTE
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BYTE
 
append(byte[], ByteBuffer) - Method in class org.apache.spark.sql.columnar.ByteArrayColumnType
 
append(JvmType, ByteBuffer) - Method in class org.apache.spark.sql.columnar.ColumnType
Appends the given value v of type T into the given ByteBuffer.
append(Row, int, ByteBuffer) - Method in class org.apache.spark.sql.columnar.ColumnType
Appends row(ordinal) of type T into the given ByteBuffer.
append(Date, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.DATE
 
append(double, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.DOUBLE
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.DOUBLE
 
append(float, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.FLOAT
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.FLOAT
 
append(int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.INT
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.INT
 
append(long, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.LONG
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.LONG
 
append(short, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.SHORT
 
append(Row, int, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.SHORT
 
append(String, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.STRING
 
append(Timestamp, ByteBuffer) - Static method in class org.apache.spark.sql.columnar.TIMESTAMP
 
append(AvroFlumeEvent) - Method in class org.apache.spark.streaming.flume.FlumeEventServer
 
appendBatch(List<AvroFlumeEvent>) - Method in class org.apache.spark.streaming.flume.FlumeEventServer
 
appendBias(Vector) - Static method in class org.apache.spark.mllib.util.MLUtils
Returns a new vector with 1.0 (bias) appended to the input vector.
appendFrom(Row, int) - Method in class org.apache.spark.sql.columnar.BasicColumnBuilder
 
appendFrom(Row, int) - Method in interface org.apache.spark.sql.columnar.ColumnBuilder
Appends row(ordinal) to the column builder.
appendFrom(Row, int) - Method in interface org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder
 
appendFrom(Row, int) - Method in interface org.apache.spark.sql.columnar.NullableColumnBuilder
 
AppendingParquetOutputFormat - Class in org.apache.spark.sql.parquet
TODO: this will be able to append to directories it created itself, not necessarily to imported ones.
AppendingParquetOutputFormat(int) - Constructor for class org.apache.spark.sql.parquet.AppendingParquetOutputFormat
 
appendReadColumns(Configuration, Seq<Integer>, Seq<String>) - Static method in class org.apache.spark.sql.hive.HiveShim
 
appId() - Method in class org.apache.spark.scheduler.ApplicationEventListener
 
appId() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
appId() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
appId() - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
appId() - Method in interface org.apache.spark.scheduler.SchedulerBackend
 
appId() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
appId() - Method in interface org.apache.spark.scheduler.TaskScheduler
 
APPLICATION_COMPLETE() - Static method in class org.apache.spark.scheduler.EventLoggingListener
 
applicationComplete() - Method in class org.apache.spark.scheduler.EventLoggingInfo
 
applicationEndFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
applicationEndToJson(SparkListenerApplicationEnd) - Static method in class org.apache.spark.util.JsonProtocol
 
ApplicationEventListener - Class in org.apache.spark.scheduler
A simple listener for application events.
ApplicationEventListener() - Constructor for class org.apache.spark.scheduler.ApplicationEventListener
 
applicationId() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
applicationId() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
applicationId() - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
applicationId() - Method in class org.apache.spark.scheduler.local.LocalBackend
 
applicationId() - Method in interface org.apache.spark.scheduler.SchedulerBackend
Get an application ID associated with the job.
applicationId() - Method in interface org.apache.spark.scheduler.TaskScheduler
Get an application ID associated with the job.
applicationId() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
applicationId() - Method in class org.apache.spark.SparkContext
 
applicationStartFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
applicationStartToJson(SparkListenerApplicationStart) - Static method in class org.apache.spark.util.JsonProtocol
 
apply(RDD<Tuple2<Object, VD>>, RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.Graph
Construct a graph from a collection of vertices and edges with attributes.
apply(RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from edges, setting referenced vertices to `defaultVertexAttr`.
apply(RDD<Tuple2<Object, VD>>, RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from vertices and edges, setting missing vertices to `defaultVertexAttr`.
apply(VertexRDD<VD>, EdgeRDD<ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from a VertexRDD and an EdgeRDD with arbitrary replicated vertices.
apply(Iterator<Tuple2<Object, VD>>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.impl.ShippableVertexPartition
Construct a `ShippableVertexPartition` from the given vertices without any routing table.
apply(Iterator<Tuple2<Object, VD>>, RoutingTablePartition, VD, ClassTag<VD>) - Static method in class org.apache.spark.graphx.impl.ShippableVertexPartition
Construct a ShippableVertexPartition from the given vertices with the specified routing table, filling in missing vertices mentioned in the routing table using defaultVal.
apply(Iterator<Tuple2<Object, VD>>, RoutingTablePartition, VD, Function2<VD, VD, VD>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.impl.ShippableVertexPartition
Construct a ShippableVertexPartition from the given vertices with the specified routing table, filling in missing vertices mentioned in the routing table using defaultVal, and merging duplicate vertex atrribute with mergeFunc.
apply(Iterator<Tuple2<Object, VD>>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.impl.VertexPartition
Construct a `VertexPartition` from the given vertices.
apply(long) - Method in class org.apache.spark.graphx.impl.VertexPartitionBase
Return the vertex attribute for the given vertex ID.
apply(Graph<VD, ED>, A, int, EdgeDirection, Function3<Object, VD, A, VD>, Function1<EdgeTriplet<VD, ED>, Iterator<Tuple2<Object, A>>>, Function2<A, A, A>, ClassTag<VD>, ClassTag<ED>, ClassTag<A>) - Static method in class org.apache.spark.graphx.Pregel
Execute a Pregel-like iterative vertex-parallel abstraction.
apply(RDD<Tuple2<Object, VD>>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a standalone VertexRDD (one that is not set up for efficient joins with an EdgeRDD) from an RDD of vertex-attribute pairs.
apply(RDD<Tuple2<Object, VD>>, EdgeRDD<?>, VD, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a VertexRDD from an RDD of vertex-attribute pairs.
apply(RDD<Tuple2<Object, VD>>, EdgeRDD<?>, VD, Function2<VD, VD, VD>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a VertexRDD from an RDD of vertex-attribute pairs.
apply(Param<T>) - Method in class org.apache.spark.ml.param.ParamMap
Gets the value of the input param or its default value if it does not exist.
apply(BinaryConfusionMatrix) - Method in interface org.apache.spark.mllib.evaluation.binary.BinaryClassificationMetricComputer
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.FalsePositiveRate
 
apply(BinaryConfusionMatrix) - Method in class org.apache.spark.mllib.evaluation.binary.FMeasure
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.Precision
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.Recall
 
apply(int) - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
apply(int, int) - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
apply(int) - Method in class org.apache.spark.mllib.linalg.DenseVector
 
apply(int, int) - Method in interface org.apache.spark.mllib.linalg.Matrix
Gets the (i, j)-th element.
apply(int, int) - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
apply(int) - Method in interface org.apache.spark.mllib.linalg.Vector
Gets the value of the ith element.
apply(int, Predict, double, boolean) - Static method in class org.apache.spark.mllib.tree.model.Node
Construct a node with nodeIndex, predict, impurity and isLeaf parameters.
apply(long, String, Option<String>, String) - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
apply(long, String, String) - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
apply(String, long, Enumeration.Value, ByteBuffer) - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate$
Alternate factory method that takes a ByteBuffer directly for the data field
apply(BlockManagerId, long[]) - Static method in class org.apache.spark.scheduler.HighlyCompressedMapStatus
 
apply(long, TaskMetrics) - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
apply(DataType) - Static method in class org.apache.spark.sql.columnar.ColumnType
 
apply(boolean, int, StorageLevel, SparkPlan, Option<String>) - Static method in class org.apache.spark.sql.columnar.InMemoryRelation
 
apply(SparkPlan) - Method in class org.apache.spark.sql.execution.AddExchange
 
apply(PythonUDF, LogicalPlan) - Static method in class org.apache.spark.sql.execution.EvaluatePython
 
apply(LogicalPlan) - Static method in class org.apache.spark.sql.execution.ExtractPythonUdfs
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.BasicOperators
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.BroadcastNestedLoopJoin
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.CartesianProduct
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.CommandStrategy
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.HashAggregation
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.HashJoin
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.InMemoryScans
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.LeftSemiJoin
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.ParquetOperations
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.execution.SparkStrategies.TakeOrdered
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog.CreateTables
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog.PreInsertionCasts
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveStrategies.DataSinks
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveStrategies.HiveCommandStrategy
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveStrategies.HiveTableScans
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveStrategies.ParquetConversion
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveStrategies.Scripts
 
apply(LogicalPlan) - Static method in class org.apache.spark.sql.sources.DataSourceStrategy
 
apply(String) - Method in class org.apache.spark.sql.sources.DDLParser
 
apply(String) - Static method in class org.apache.spark.storage.BlockId
Converts a BlockId "name" String back into a BlockId.
apply(String, String, int) - Static method in class org.apache.spark.storage.BlockManagerId
Returns a BlockManagerId for the given configuration.
apply(ObjectInput) - Static method in class org.apache.spark.storage.BlockManagerId
 
apply(boolean, boolean, boolean, boolean, int) - Static method in class org.apache.spark.storage.StorageLevel
:: DeveloperApi :: Create a new StorageLevel object without setting useOffHeap.
apply(boolean, boolean, boolean, int) - Static method in class org.apache.spark.storage.StorageLevel
:: DeveloperApi :: Create a new StorageLevel object.
apply(int, int) - Static method in class org.apache.spark.storage.StorageLevel
:: DeveloperApi :: Create a new StorageLevel object from its integer representation.
apply(ObjectInput) - Static method in class org.apache.spark.storage.StorageLevel
:: DeveloperApi :: Read StorageLevel object from ObjectInput stream.
apply(long) - Static method in class org.apache.spark.streaming.Milliseconds
 
apply(long) - Static method in class org.apache.spark.streaming.Minutes
 
apply(long) - Static method in class org.apache.spark.streaming.Seconds
 
apply(I, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.CompletionIterator
 
apply(Traversable<Object>) - Static method in class org.apache.spark.util.Distribution
 
apply(InputStream, File, SparkConf) - Static method in class org.apache.spark.util.logging.FileAppender
Create the right appender based on Spark configuration
apply(TraversableOnce<Object>) - Static method in class org.apache.spark.util.StatCounter
Build a StatCounter from a list of values.
apply(Seq<Object>) - Static method in class org.apache.spark.util.StatCounter
Build a StatCounter from a list of values passed as variable-length arguments.
apply(A) - Method in class org.apache.spark.util.TimeStampedHashMap
 
apply(A) - Method in class org.apache.spark.util.TimeStampedWeakValueHashMap
 
apply(int) - Method in class org.apache.spark.util.Vector
 
applySchema(JavaRDD<?>, Class<?>) - Method in class org.apache.spark.sql.api.java.JavaSQLContext
Applies a schema to an RDD of Java Beans.
applySchema(JavaRDD<Row>, StructType) - Method in class org.apache.spark.sql.api.java.JavaSQLContext
:: DeveloperApi :: Creates a JavaSchemaRDD from an RDD containing Rows by applying a schema to this RDD.
applySchema(RDD<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
:: DeveloperApi :: Creates a SchemaRDD from an RDD containing Rows by applying a schema to this RDD.
applySchemaToPythonRDD(RDD<Object[]>, String) - Method in class org.apache.spark.sql.SQLContext
Apply a schema defined by the schemaString to an RDD.
applySchemaToPythonRDD(RDD<Object[]>, StructType) - Method in class org.apache.spark.sql.SQLContext
Apply a schema defined by the schema to an RDD.
appName() - Method in class org.apache.spark.api.java.JavaSparkContext
 
appName() - Method in class org.apache.spark.scheduler.ApplicationEventListener
 
appName() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
appName() - Method in class org.apache.spark.SparkContext
 
appName() - Method in class org.apache.spark.ui.SparkUI
 
appName() - Method in class org.apache.spark.ui.SparkUITab
 
ApproxHist() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
ApproximateActionListener<T,U,R> - Class in org.apache.spark.partial
A JobListener for an approximate single-result action, such as count() or non-parallel reduce().
ApproximateActionListener(RDD<T>, Function2<TaskContext, Iterator<T>, U>, ApproximateEvaluator<U, R>, long) - Constructor for class org.apache.spark.partial.ApproximateActionListener
 
ApproximateEvaluator<U,R> - Interface in org.apache.spark.partial
An object that computes a function incrementally by merging in results of type U from multiple tasks.
appUIAddress() - Method in class org.apache.spark.ui.SparkUI
 
appUIHostPort() - Method in class org.apache.spark.ui.SparkUI
Return the application UI host:port.
AreaUnderCurve - Class in org.apache.spark.mllib.evaluation
Computes the area under the curve (AUC) using the trapezoidal rule.
AreaUnderCurve() - Constructor for class org.apache.spark.mllib.evaluation.AreaUnderCurve
 
areaUnderPR() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Computes the area under the precision-recall curve.
areaUnderROC() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Computes the area under the receiver operating characteristic (ROC) curve.
areBoundsEmpty() - Method in class org.apache.spark.util.random.AcceptanceResult
 
argString() - Method in class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
arr() - Method in class org.apache.spark.rdd.PartitionGroup
 
ARRAY() - Static method in class org.apache.spark.sql.hive.HiveQl
 
ARRAY_CONTAINS_NULL_BAG_SCHEMA_NAME() - Static method in class org.apache.spark.sql.parquet.CatalystConverter
 
ARRAY_ELEMENTS_SCHEMA_NAME() - Static method in class org.apache.spark.sql.parquet.CatalystConverter
 
arrayBuffer() - Method in class org.apache.spark.streaming.receiver.ArrayBufferBlock
 
ArrayBufferBlock - Class in org.apache.spark.streaming.receiver
class representing a block received as an ArrayBuffer
ArrayBufferBlock(ArrayBuffer<?>) - Constructor for class org.apache.spark.streaming.receiver.ArrayBufferBlock
 
ArrayType - Class in org.apache.spark.sql.api.java
The data type representing Lists.
ArrayValues - Class in org.apache.spark.storage
 
ArrayValues(Object[]) - Constructor for class org.apache.spark.storage.ArrayValues
 
as(Symbol) - Method in class org.apache.spark.sql.SchemaRDD
Applies a qualifier to the attributes of this relation.
asIterator() - Method in class org.apache.spark.serializer.DeserializationStream
Read the elements of this stream through an iterator.
asJavaDataType(DataType) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Returns the equivalent DataType in Java for the given DataType in Scala.
asJavaStructField(StructField) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Returns the equivalent StructField in Scala for the given StructField in Java.
askSlaves() - Method in class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
askSlaves() - Method in class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds
 
askTimeout(SparkConf) - Static method in class org.apache.spark.util.AkkaUtils
Returns the default Spark timeout to use for Akka ask operations.
askWithReply(Object, ActorRef, FiniteDuration) - Static method in class org.apache.spark.util.AkkaUtils
Send a message to the given actor and get its result within a default timeout, or throw a SparkException if this fails.
askWithReply(Object, ActorRef, int, int, FiniteDuration) - Static method in class org.apache.spark.util.AkkaUtils
Send a message to the given actor and get its result within a default timeout, or throw a SparkException if this fails even after the specified number of retries.
asRDDId() - Method in class org.apache.spark.storage.BlockId
 
asScalaDataType(DataType) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Returns the equivalent DataType in Scala for the given DataType in Java.
asScalaStructField(StructField) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Returns the equivalent StructField in Scala for the given StructField in Java.
assertValid() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
Check validity of parameters.
assertValid() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
Check validity of parameters.
assertValid() - Method in class org.apache.spark.rdd.BlockRDD
Check if this BlockRDD is valid.
AsyncRDDActions<T> - Class in org.apache.spark.rdd
A set of asynchronous RDD actions available through an implicit conversion.
AsyncRDDActions(RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.AsyncRDDActions
 
attachExecutor(ReceiverSupervisor) - Method in class org.apache.spark.streaming.receiver.Receiver
Attach Network Receiver executor to this receiver.
attachHandler(ServletContextHandler) - Method in class org.apache.spark.ui.WebUI
Attach a handler to this UI.
attachListener(CleanerListener) - Method in class org.apache.spark.ContextCleaner
Attach a listener object to get information of when objects are cleaned.
attachPage(WebUIPage) - Method in class org.apache.spark.ui.WebUI
Attach a page to this UI.
attachPage(WebUIPage) - Method in class org.apache.spark.ui.WebUITab
Attach a page to this tab.
attachTab(WebUITab) - Method in class org.apache.spark.ui.WebUI
Attach a tab to this UI, along with all of its attached pages.
attempt() - Method in class org.apache.spark.scheduler.TaskInfo
 
attempt() - Method in class org.apache.spark.scheduler.TaskSet
 
attemptId() - Method in class org.apache.spark.scheduler.Stage
 
attemptId() - Method in class org.apache.spark.scheduler.StageInfo
 
attemptId() - Method in class org.apache.spark.TaskContext
 
attemptId() - Method in class org.apache.spark.TaskContextImpl
 
attr() - Method in class org.apache.spark.graphx.Edge
 
attr() - Method in class org.apache.spark.graphx.EdgeContext
The attribute associated with the edge.
attr() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
attr() - Method in class org.apache.spark.graphx.impl.EdgeWithLocalIds
 
attribute() - Method in class org.apache.spark.sql.sources.EqualTo
 
attribute() - Method in class org.apache.spark.sql.sources.GreaterThan
 
attribute() - Method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
attribute() - Method in class org.apache.spark.sql.sources.In
 
attribute() - Method in class org.apache.spark.sql.sources.LessThan
 
attribute() - Method in class org.apache.spark.sql.sources.LessThanOrEqual
 
attributeMap() - Method in class org.apache.spark.sql.hive.MetastoreRelation
An attribute map that can be used to lookup original attributes based on expression id.
attributeMap() - Method in class org.apache.spark.sql.sources.LogicalRelation
Used to lookup original attribute capitalization
attributes() - Method in class org.apache.spark.sql.columnar.InMemoryColumnarTableScan
 
attributes() - Method in class org.apache.spark.sql.hive.execution.HiveTableScan
 
attributes() - Method in class org.apache.spark.sql.hive.MetastoreRelation
Non-partitionKey attributes
attributes() - Method in class org.apache.spark.sql.parquet.ParquetTableScan
 
attributes() - Method in class org.apache.spark.sql.parquet.RowWriteSupport
 
attrs() - Method in class org.apache.spark.graphx.impl.VertexAttributeBlock
 
autoBroadcastJoinThreshold() - Method in interface org.apache.spark.sql.SQLConf
Upper bound on the sizes (in bytes) of the tables qualified for the auto conversion to a broadcast value during the physical executions of join operations.
Average() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
AVG() - Static method in class org.apache.spark.sql.hive.HiveQl
 
awaitResult() - Method in class org.apache.spark.partial.ApproximateActionListener
Waits for up to timeout milliseconds since the listener was created and then returns a PartialResult with the result so far.
awaitResult() - Method in class org.apache.spark.scheduler.JobWaiter
 
awaitTermination() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Wait for the execution to stop.
awaitTermination(long) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Wait for the execution to stop.
awaitTermination() - Method in class org.apache.spark.streaming.receiver.ReceiverSupervisor
Wait the thread until the supervisor is stopped
awaitTermination() - Method in class org.apache.spark.streaming.StreamingContext
Wait for the execution to stop.
awaitTermination(long) - Method in class org.apache.spark.streaming.StreamingContext
Wait for the execution to stop.
awaitTermination() - Method in class org.apache.spark.util.logging.FileAppender
Wait for the appender to stop appending, either because input stream is closed or because of any error in appending
axpy(double, Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
y += a * x

B

backend() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
BaggedPoint<Datum> - Class in org.apache.spark.mllib.tree.impl
Internal representation of a datapoint which belongs to several subsamples of the same dataset, particularly for bagging (e.g., for random forests).
BaggedPoint(Datum, double[]) - Constructor for class org.apache.spark.mllib.tree.impl.BaggedPoint
 
base() - Method in class org.apache.spark.sql.hive.HiveUdafFunction
 
baseDir() - Method in class org.apache.spark.HttpFileServer
 
baseLogicalPlan() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
 
baseLogicalPlan() - Method in class org.apache.spark.sql.SchemaRDD
 
baseLogicalPlan() - Method in interface org.apache.spark.sql.SchemaRDDLike
 
baseOn(ParamPair<?>...) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
baseOn(ParamMap) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
baseOn(Seq<ParamPair<?>>) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
basePath() - Method in class org.apache.spark.ui.SparkUI
 
basePath() - Method in class org.apache.spark.ui.WebUITab
 
BaseRelation - Class in org.apache.spark.sql.sources
::DeveloperApi:: Represents a collection of tuples with a known schema.
BaseRelation() - Constructor for class org.apache.spark.sql.sources.BaseRelation
 
baseRelationToSchemaRDD(BaseRelation) - Method in class org.apache.spark.sql.api.java.JavaSQLContext
 
baseRelationToSchemaRDD(BaseRelation) - Method in class org.apache.spark.sql.SQLContext
 
baseSchemaRDD() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
 
baseSchemaRDD() - Method in class org.apache.spark.sql.SchemaRDD
 
baseSchemaRDD() - Method in interface org.apache.spark.sql.SchemaRDDLike
 
BasicColumnAccessor<T extends org.apache.spark.sql.catalyst.types.DataType,JvmType> - Class in org.apache.spark.sql.columnar
 
BasicColumnAccessor(ByteBuffer, ColumnType<T, JvmType>) - Constructor for class org.apache.spark.sql.columnar.BasicColumnAccessor
 
BasicColumnBuilder<T extends org.apache.spark.sql.catalyst.types.DataType,JvmType> - Class in org.apache.spark.sql.columnar
 
BasicColumnBuilder(ColumnStats, ColumnType<T, JvmType>) - Constructor for class org.apache.spark.sql.columnar.BasicColumnBuilder
 
BasicOperators() - Method in class org.apache.spark.sql.execution.SparkStrategies
 
basicSparkPage(Function0<Seq<Node>>, String) - Static method in class org.apache.spark.ui.UIUtils
Returns a page with the spark css/js and a simple format.
BatchAllocationEvent - Class in org.apache.spark.streaming.scheduler
 
BatchAllocationEvent(Time, AllocatedBlocks) - Constructor for class org.apache.spark.streaming.scheduler.BatchAllocationEvent
 
BatchCleanupEvent - Class in org.apache.spark.streaming.scheduler
 
BatchCleanupEvent(Seq<Time>) - Constructor for class org.apache.spark.streaming.scheduler.BatchCleanupEvent
 
batchDuration() - Method in class org.apache.spark.streaming.DStreamGraph
 
batchDuration() - Method in class org.apache.spark.streaming.ui.StreamingJobProgressListener
 
BATCHES() - Static method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
BatchInfo - Class in org.apache.spark.streaming.scheduler
:: DeveloperApi :: Class having information on completed batches.
BatchInfo(Time, Map<Object, ReceivedBlockInfo[]>, long, Option<Object>, Option<Object>) - Constructor for class org.apache.spark.streaming.scheduler.BatchInfo
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
batchInfos() - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
BatchPythonEvaluation - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Uses PythonRDD to evaluate a PythonUDF, one partition of tuples at a time.
BatchPythonEvaluation(PythonUDF, Seq<Attribute>, SparkPlan) - Constructor for class org.apache.spark.sql.execution.BatchPythonEvaluation
 
batchSize() - Method in class org.apache.spark.sql.columnar.InMemoryRelation
 
batchTime() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
batchTimeToSelectedFiles() - Method in class org.apache.spark.streaming.dstream.FileInputDStream
 
BeginEvent - Class in org.apache.spark.scheduler
 
BeginEvent(Task<?>, TaskInfo) - Constructor for class org.apache.spark.scheduler.BeginEvent
 
beginTime() - Method in class org.apache.spark.streaming.Interval
 
benchmark(int) - Static method in class org.apache.spark.util.random.XORShiftRandom
 
BernoulliCellSampler<T> - Class in org.apache.spark.util.random
:: DeveloperApi :: A sampler based on Bernoulli trials for partitioning a data sequence.
BernoulliCellSampler(double, double, boolean) - Constructor for class org.apache.spark.util.random.BernoulliCellSampler
 
BernoulliSampler<T> - Class in org.apache.spark.util.random
:: DeveloperApi :: A sampler based on Bernoulli trials.
BernoulliSampler(double, ClassTag<T>) - Constructor for class org.apache.spark.util.random.BernoulliSampler
 
bestModel() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
beta() - Method in class org.apache.spark.mllib.evaluation.binary.FMeasure
 
BETWEEN() - Static method in class org.apache.spark.sql.hive.HiveQl
 
BigDecimalSerializer - Class in org.apache.spark.sql.execution
 
BigDecimalSerializer() - Constructor for class org.apache.spark.sql.execution.BigDecimalSerializer
 
Bin - Class in org.apache.spark.mllib.tree.model
Used for "binning" the feature values for faster best split calculation.
Bin(Split, Split, Enumeration.Value, double) - Constructor for class org.apache.spark.mllib.tree.model.Bin
 
BINARY - Class in org.apache.spark.sql.columnar
 
BINARY() - Constructor for class org.apache.spark.sql.columnar.BINARY
 
BinaryClassificationEvaluator - Class in org.apache.spark.ml.evaluation
:: AlphaComponent :: Evaluator for binary classification, which expects two input columns: score and label.
BinaryClassificationEvaluator() - Constructor for class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
BinaryClassificationMetricComputer - Interface in org.apache.spark.mllib.evaluation.binary
Trait for a binary classification evaluation metric computer.
BinaryClassificationMetrics - Class in org.apache.spark.mllib.evaluation
:: Experimental :: Evaluator for binary classification.
BinaryClassificationMetrics(RDD<Tuple2<Object, Object>>) - Constructor for class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
 
BinaryColumnAccessor - Class in org.apache.spark.sql.columnar
 
BinaryColumnAccessor(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.BinaryColumnAccessor
 
BinaryColumnBuilder - Class in org.apache.spark.sql.columnar
 
BinaryColumnBuilder() - Constructor for class org.apache.spark.sql.columnar.BinaryColumnBuilder
 
BinaryColumnStats - Class in org.apache.spark.sql.columnar
 
BinaryColumnStats() - Constructor for class org.apache.spark.sql.columnar.BinaryColumnStats
 
BinaryConfusionMatrix - Interface in org.apache.spark.mllib.evaluation.binary
Trait for a binary confusion matrix.
BinaryConfusionMatrixImpl - Class in org.apache.spark.mllib.evaluation.binary
Implementation of BinaryConfusionMatrix.
BinaryConfusionMatrixImpl(BinaryLabelCounter, BinaryLabelCounter) - Constructor for class org.apache.spark.mllib.evaluation.binary.BinaryConfusionMatrixImpl
 
BinaryFileRDD<T> - Class in org.apache.spark.rdd
 
BinaryFileRDD(SparkContext, Class<? extends StreamFileInputFormat<T>>, Class<String>, Class<T>, Configuration, int) - Constructor for class org.apache.spark.rdd.BinaryFileRDD
 
binaryFiles(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a directory of binary files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI as a byte array.
binaryFiles(String) - Method in class org.apache.spark.api.java.JavaSparkContext
:: Experimental ::
binaryFiles(String, int) - Method in class org.apache.spark.SparkContext
:: Experimental ::
BinaryLabelCounter - Class in org.apache.spark.mllib.evaluation.binary
A counter for positives and negatives.
BinaryLabelCounter(long, long) - Constructor for class org.apache.spark.mllib.evaluation.binary.BinaryLabelCounter
 
binaryLabelValidator() - Static method in class org.apache.spark.mllib.util.DataValidators
Function to check if labels used for classification are either zero or one.
BinaryNode - Interface in org.apache.spark.sql.execution
 
binaryRecords(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
:: Experimental ::
binaryRecords(String, int, Configuration) - Method in class org.apache.spark.SparkContext
:: Experimental ::
BinaryType - Class in org.apache.spark.sql.api.java
The data type representing byte[] values.
BinaryType - Static variable in class org.apache.spark.sql.api.java.DataType
Gets the BinaryType object.
bind() - Method in class org.apache.spark.ui.WebUI
Bind to the HTTP server behind this web interface.
binnedFeatures() - Method in class org.apache.spark.mllib.tree.impl.TreePoint
 
BinomialBounds - Class in org.apache.spark.util.random
Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact sample size with high confidence when sampling without replacement.
BinomialBounds() - Constructor for class org.apache.spark.util.random.BinomialBounds
 
BITS_PER_LONG() - Static method in class org.apache.spark.sql.columnar.compression.BooleanBitSet
 
BLAS - Class in org.apache.spark.mllib.linalg
BLAS routines for MLlib's vectors and matrices.
BLAS() - Constructor for class org.apache.spark.mllib.linalg.BLAS
 
BLOCK_MANAGER() - Static method in class org.apache.spark.util.MetadataCleanerType
 
BlockAdditionEvent - Class in org.apache.spark.streaming.scheduler
 
BlockAdditionEvent(ReceivedBlockInfo) - Constructor for class org.apache.spark.streaming.scheduler.BlockAdditionEvent
 
BlockException - Exception in org.apache.spark.storage
 
BlockException(BlockId, String) - Constructor for exception org.apache.spark.storage.BlockException
 
BlockGenerator - Class in org.apache.spark.streaming.receiver
Generates batches of objects received by a Receiver and puts them into appropriately named blocks at regular intervals.
BlockGenerator(BlockGeneratorListener, int, SparkConf) - Constructor for class org.apache.spark.streaming.receiver.BlockGenerator
 
BlockGeneratorListener - Interface in org.apache.spark.streaming.receiver
Listener object for BlockGenerator events
blockId() - Method in class org.apache.spark.rdd.BlockRDDPartition
 
blockId() - Method in class org.apache.spark.scheduler.IndirectTaskResult
 
blockId() - Method in exception org.apache.spark.storage.BlockException
 
BlockId - Class in org.apache.spark.storage
:: DeveloperApi :: Identifies a particular Block of data, usually associated with a single file.
BlockId() - Constructor for class org.apache.spark.storage.BlockId
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetLocations
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveBlock
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
blockId() - Method in class org.apache.spark.storage.BlockObjectWriter
 
blockId() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.FailureFetchResult
 
blockId() - Method in interface org.apache.spark.storage.ShuffleBlockFetcherIterator.FetchResult
 
blockId() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.SuccessFetchResult
 
blockId() - Method in class org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDDPartition
 
blockId() - Method in class org.apache.spark.streaming.receiver.BlockManagerBasedStoreResult
 
blockId() - Method in interface org.apache.spark.streaming.receiver.ReceivedBlockStoreResult
 
blockId() - Method in class org.apache.spark.streaming.receiver.WriteAheadLogBasedStoreResult
 
blockIds() - Method in class org.apache.spark.rdd.BlockRDD
 
blockIds() - Method in class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds
 
blockIdsToBlockManagers(BlockId[], SparkEnv, BlockManagerMaster) - Static method in class org.apache.spark.storage.BlockManager
 
blockIdsToExecutorIds(BlockId[], SparkEnv, BlockManagerMaster) - Static method in class org.apache.spark.storage.BlockManager
 
blockIdsToHosts(BlockId[], SparkEnv, BlockManagerMaster) - Static method in class org.apache.spark.storage.BlockManager
 
blockifyObject(T, int, Serializer, Option<CompressionCodec>, ClassTag<T>) - Static method in class org.apache.spark.broadcast.TorrentBroadcast
 
BlockInfo - Class in org.apache.spark.storage
 
BlockInfo(StorageLevel, boolean) - Constructor for class org.apache.spark.storage.BlockInfo
 
blockManager() - Method in class org.apache.spark.SparkEnv
 
BlockManager - Class in org.apache.spark.storage
Manager running on every node (driver and executors) which provides interfaces for putting and retrieving blocks both locally and remotely into various stores (memory, disk, and off-heap).
BlockManager(String, ActorSystem, BlockManagerMaster, Serializer, long, SparkConf, MapOutputTracker, ShuffleManager, BlockTransferService, SecurityManager, int) - Constructor for class org.apache.spark.storage.BlockManager
 
BlockManager(String, ActorSystem, BlockManagerMaster, Serializer, SparkConf, MapOutputTracker, ShuffleManager, BlockTransferService, SecurityManager, int) - Constructor for class org.apache.spark.storage.BlockManager
Construct a BlockManager with a memory limit set based on system properties.
blockManager() - Method in class org.apache.spark.storage.BlockManagerSource
 
blockManager() - Method in class org.apache.spark.storage.BlockStore
 
blockManagerAddedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerAddedToJson(SparkListenerBlockManagerAdded) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockManagerBasedBlockHandler - Class in org.apache.spark.streaming.receiver
Implementation of a ReceivedBlockHandler which stores the received blocks into a block manager with the specified storage level.
BlockManagerBasedBlockHandler(BlockManager, StorageLevel) - Constructor for class org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler
 
BlockManagerBasedStoreResult - Class in org.apache.spark.streaming.receiver
Implementation of ReceivedBlockStoreResult that stores the metadata related to storage of blocks using BlockManagerBasedBlockHandler
BlockManagerBasedStoreResult(StreamBlockId) - Constructor for class org.apache.spark.streaming.receiver.BlockManagerBasedStoreResult
 
blockManagerId() - Method in class org.apache.spark.Heartbeat
 
blockManagerId() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
blockManagerId() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManager
 
BlockManagerId - Class in org.apache.spark.storage
:: DeveloperApi :: This class represent an unique identifier for a BlockManager.
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerInfo
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetPeers
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
blockManagerId() - Method in class org.apache.spark.storage.StorageStatus
 
blockManagerIdCache() - Static method in class org.apache.spark.storage.BlockManagerId
 
blockManagerIdFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerIds() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
blockManagerIdToJson(BlockManagerId) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockManagerInfo - Class in org.apache.spark.storage
 
BlockManagerInfo(BlockManagerId, long, long, ActorRef) - Constructor for class org.apache.spark.storage.BlockManagerInfo
 
BlockManagerMaster - Class in org.apache.spark.storage
 
BlockManagerMaster(ActorRef, SparkConf, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMaster
 
BlockManagerMasterActor - Class in org.apache.spark.storage
BlockManagerMasterActor is an actor on the master node to track statuses of all slaves' block managers.
BlockManagerMasterActor(boolean, SparkConf, LiveListenerBus) - Constructor for class org.apache.spark.storage.BlockManagerMasterActor
 
BlockManagerMessages - Class in org.apache.spark.storage
 
BlockManagerMessages() - Constructor for class org.apache.spark.storage.BlockManagerMessages
 
BlockManagerMessages.BlockManagerHeartbeat - Class in org.apache.spark.storage
 
BlockManagerMessages.BlockManagerHeartbeat(BlockManagerId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat
 
BlockManagerMessages.BlockManagerHeartbeat$ - Class in org.apache.spark.storage
 
BlockManagerMessages.BlockManagerHeartbeat$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat$
 
BlockManagerMessages.ExpireDeadHosts$ - Class in org.apache.spark.storage
 
BlockManagerMessages.ExpireDeadHosts$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.ExpireDeadHosts$
 
BlockManagerMessages.GetActorSystemHostPortForExecutor - Class in org.apache.spark.storage
 
BlockManagerMessages.GetActorSystemHostPortForExecutor(String) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetActorSystemHostPortForExecutor
 
BlockManagerMessages.GetActorSystemHostPortForExecutor$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetActorSystemHostPortForExecutor$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetActorSystemHostPortForExecutor$
 
BlockManagerMessages.GetBlockStatus - Class in org.apache.spark.storage
 
BlockManagerMessages.GetBlockStatus(BlockId, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
BlockManagerMessages.GetBlockStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetBlockStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus$
 
BlockManagerMessages.GetLocations - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocations(BlockId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocations
 
BlockManagerMessages.GetLocations$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocations$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocations$
 
BlockManagerMessages.GetLocationsMultipleBlockIds - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsMultipleBlockIds(BlockId[]) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds
 
BlockManagerMessages.GetLocationsMultipleBlockIds$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsMultipleBlockIds$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds$
 
BlockManagerMessages.GetMatchingBlockIds - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMatchingBlockIds(Function1<BlockId, Object>, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds
 
BlockManagerMessages.GetMatchingBlockIds$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMatchingBlockIds$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds$
 
BlockManagerMessages.GetMemoryStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMemoryStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMemoryStatus$
 
BlockManagerMessages.GetPeers - Class in org.apache.spark.storage
 
BlockManagerMessages.GetPeers(BlockManagerId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetPeers
 
BlockManagerMessages.GetPeers$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetPeers$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetPeers$
 
BlockManagerMessages.GetStorageStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetStorageStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetStorageStatus$
 
BlockManagerMessages.RegisterBlockManager - Class in org.apache.spark.storage
 
BlockManagerMessages.RegisterBlockManager(BlockManagerId, long, ActorRef) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
BlockManagerMessages.RegisterBlockManager$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RegisterBlockManager$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager$
 
BlockManagerMessages.RemoveBlock - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBlock(BlockId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBlock
 
BlockManagerMessages.RemoveBlock$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBlock$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBlock$
 
BlockManagerMessages.RemoveBroadcast - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBroadcast(long, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast
 
BlockManagerMessages.RemoveBroadcast$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBroadcast$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast$
 
BlockManagerMessages.RemoveExecutor - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveExecutor(String) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor
 
BlockManagerMessages.RemoveExecutor$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveExecutor$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor$
 
BlockManagerMessages.RemoveRdd - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveRdd(int) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveRdd
 
BlockManagerMessages.RemoveRdd$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveRdd$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveRdd$
 
BlockManagerMessages.RemoveShuffle - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveShuffle(int) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle
 
BlockManagerMessages.RemoveShuffle$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveShuffle$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle$
 
BlockManagerMessages.StopBlockManagerMaster$ - Class in org.apache.spark.storage
 
BlockManagerMessages.StopBlockManagerMaster$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.StopBlockManagerMaster$
 
BlockManagerMessages.ToBlockManagerMaster - Interface in org.apache.spark.storage
 
BlockManagerMessages.ToBlockManagerSlave - Interface in org.apache.spark.storage
 
BlockManagerMessages.UpdateBlockInfo - Class in org.apache.spark.storage
 
BlockManagerMessages.UpdateBlockInfo(BlockManagerId, BlockId, StorageLevel, long, long, long) - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
BlockManagerMessages.UpdateBlockInfo() - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
BlockManagerMessages.UpdateBlockInfo$ - Class in org.apache.spark.storage
 
BlockManagerMessages.UpdateBlockInfo$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo$
 
blockManagerRemovedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerRemovedToJson(SparkListenerBlockManagerRemoved) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockManagerSlaveActor - Class in org.apache.spark.storage
An actor to take commands from the master to execute options.
BlockManagerSlaveActor(BlockManager, MapOutputTracker) - Constructor for class org.apache.spark.storage.BlockManagerSlaveActor
 
BlockManagerSource - Class in org.apache.spark.storage
 
BlockManagerSource(BlockManager) - Constructor for class org.apache.spark.storage.BlockManagerSource
 
BlockNotFoundException - Exception in org.apache.spark.storage
 
BlockNotFoundException(String) - Constructor for exception org.apache.spark.storage.BlockNotFoundException
 
BlockObjectWriter - Class in org.apache.spark.storage
An interface for writing JVM objects to some underlying storage.
BlockObjectWriter(BlockId) - Constructor for class org.apache.spark.storage.BlockObjectWriter
 
blockPushingThread() - Method in class org.apache.spark.streaming.dstream.RawNetworkReceiver
 
BlockRDD<T> - Class in org.apache.spark.rdd
 
BlockRDD(SparkContext, BlockId[], ClassTag<T>) - Constructor for class org.apache.spark.rdd.BlockRDD
 
BlockRDDPartition - Class in org.apache.spark.rdd
 
BlockRDDPartition(BlockId, int) - Constructor for class org.apache.spark.rdd.BlockRDDPartition
 
BlockResult - Class in org.apache.spark.storage
 
BlockResult(Iterator<Object>, Enumeration.Value, long) - Constructor for class org.apache.spark.storage.BlockResult
 
blocks() - Method in class org.apache.spark.storage.BlockManagerInfo
 
blocks() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.FetchRequest
 
blocks() - Method in class org.apache.spark.storage.StorageStatus
Return the blocks stored in this block manager.
BlockStatus - Class in org.apache.spark.storage
 
BlockStatus(StorageLevel, long, long, long) - Constructor for class org.apache.spark.storage.BlockStatus
 
blockStatusFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockStatusToJson(BlockStatus) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockStore - Class in org.apache.spark.storage
Abstract class to store blocks.
BlockStore(BlockManager) - Constructor for class org.apache.spark.storage.BlockStore
 
blockStoreResult() - Method in class org.apache.spark.streaming.scheduler.ReceivedBlockInfo
 
blockTransferService() - Method in class org.apache.spark.SparkEnv
 
BlockValues - Interface in org.apache.spark.storage
 
bmAddress() - Method in class org.apache.spark.FetchFailed
 
BOOLEAN - Class in org.apache.spark.sql.columnar
 
BOOLEAN() - Constructor for class org.apache.spark.sql.columnar.BOOLEAN
 
BooleanBitSet - Class in org.apache.spark.sql.columnar.compression
 
BooleanBitSet() - Constructor for class org.apache.spark.sql.columnar.compression.BooleanBitSet
 
BooleanBitSet.Decoder - Class in org.apache.spark.sql.columnar.compression
 
BooleanBitSet.Decoder(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.compression.BooleanBitSet.Decoder
 
BooleanBitSet.Encoder - Class in org.apache.spark.sql.columnar.compression
 
BooleanBitSet.Encoder() - Constructor for class org.apache.spark.sql.columnar.compression.BooleanBitSet.Encoder
 
BooleanColumnAccessor - Class in org.apache.spark.sql.columnar
 
BooleanColumnAccessor(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.BooleanColumnAccessor
 
BooleanColumnBuilder - Class in org.apache.spark.sql.columnar
 
BooleanColumnBuilder() - Constructor for class org.apache.spark.sql.columnar.BooleanColumnBuilder
 
BooleanColumnStats - Class in org.apache.spark.sql.columnar
 
BooleanColumnStats() - Constructor for class org.apache.spark.sql.columnar.BooleanColumnStats
 
BooleanParam - Class in org.apache.spark.ml.param
Specialized version of Param[Boolean] for Java.
BooleanParam(Params, String, String, Option<Object>) - Constructor for class org.apache.spark.ml.param.BooleanParam
 
BooleanType - Class in org.apache.spark.sql.api.java
The data type representing boolean and Boolean values.
BooleanType - Static variable in class org.apache.spark.sql.api.java.DataType
Gets the BooleanType object.
booleanWritableConverter() - Static method in class org.apache.spark.SparkContext
 
boolToBoolWritable(boolean) - Static method in class org.apache.spark.SparkContext
 
BoostingStrategy - Class in org.apache.spark.mllib.tree.configuration
:: Experimental :: Configuration options for GradientBoostedTrees.
BoostingStrategy(Strategy, Loss, int, double) - Constructor for class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
Both() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges originating from *and* arriving at a vertex of interest.
BoundedDouble - Class in org.apache.spark.partial
:: Experimental :: A Double value with error bars and associated confidence.
BoundedDouble(double, double, double, double) - Constructor for class org.apache.spark.partial.BoundedDouble
 
BoundedPriorityQueue<A> - Class in org.apache.spark.util
Bounded priority queue.
BoundedPriorityQueue(int, Ordering<A>) - Constructor for class org.apache.spark.util.BoundedPriorityQueue
 
boundGenerator() - Method in class org.apache.spark.sql.execution.Generate
 
boundPort() - Method in class org.apache.spark.ui.ServerInfo
 
boundPort() - Method in class org.apache.spark.ui.WebUI
Return the actual port to which this server is bound.
broadcast(T) - Method in class org.apache.spark.api.java.JavaSparkContext
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions.
Broadcast<T> - Class in org.apache.spark.broadcast
A broadcast variable.
Broadcast(long, ClassTag<T>) - Constructor for class org.apache.spark.broadcast.Broadcast
 
broadcast(T, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions.
broadcast() - Method in class org.apache.spark.sql.execution.joins.LeftSemiJoinBNL
 
BROADCAST() - Static method in class org.apache.spark.storage.BlockId
 
BROADCAST_VARS() - Static method in class org.apache.spark.util.MetadataCleanerType
 
BroadcastBlockId - Class in org.apache.spark.storage
 
BroadcastBlockId(long, String) - Constructor for class org.apache.spark.storage.BroadcastBlockId
 
broadcastCleaned(long) - Method in interface org.apache.spark.CleanerListener
 
broadcastedConf() - Method in class org.apache.spark.rdd.CheckpointRDD
 
BroadcastFactory - Interface in org.apache.spark.broadcast
:: DeveloperApi :: An interface for all the broadcast implementations in Spark (to allow multiple broadcast implementations).
BroadcastHashJoin - Class in org.apache.spark.sql.execution.joins
:: DeveloperApi :: Performs an inner hash join of two child relations.
BroadcastHashJoin(Seq<Expression>, Seq<Expression>, org.apache.spark.sql.execution.joins.BuildSide, SparkPlan, SparkPlan) - Constructor for class org.apache.spark.sql.execution.joins.BroadcastHashJoin
 
broadcastId() - Method in class org.apache.spark.CleanBroadcast
 
broadcastId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast
 
broadcastId() - Method in class org.apache.spark.storage.BroadcastBlockId
 
BroadcastManager - Class in org.apache.spark.broadcast
 
BroadcastManager(boolean, SparkConf, SecurityManager) - Constructor for class org.apache.spark.broadcast.BroadcastManager
 
broadcastManager() - Method in class org.apache.spark.SparkEnv
 
BroadcastNestedLoopJoin - Class in org.apache.spark.sql.execution.joins
:: DeveloperApi ::
BroadcastNestedLoopJoin(SparkPlan, SparkPlan, org.apache.spark.sql.execution.joins.BuildSide, JoinType, Option<Expression>) - Constructor for class org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoin
 
BroadcastNestedLoopJoin() - Method in class org.apache.spark.sql.execution.SparkStrategies
 
broadcastVars() - Method in class org.apache.spark.sql.execution.PythonUDF
 
buf() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.SuccessFetchResult
 
buffer() - Method in class org.apache.spark.storage.ArrayValues
 
buffer() - Method in class org.apache.spark.storage.ByteBufferValues
 
buffer() - Method in class org.apache.spark.util.SerializableBuffer
 
buffers() - Method in class org.apache.spark.sql.columnar.CachedBatch
 
build() - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Builds and returns all combinations of parameters specified by the param grid.
build(Node[]) - Method in class org.apache.spark.mllib.tree.model.Node
build the left node and right nodes if not leaf
build() - Method in class org.apache.spark.sql.api.java.MetadataBuilder
 
build() - Method in class org.apache.spark.sql.columnar.BasicColumnBuilder
 
build() - Method in interface org.apache.spark.sql.columnar.ColumnBuilder
Returns the final columnar byte buffer.
build() - Method in interface org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder
 
build() - Method in interface org.apache.spark.sql.columnar.NullableColumnBuilder
 
buildFilter() - Method in class org.apache.spark.sql.columnar.InMemoryColumnarTableScan
 
buildKeys() - Method in interface org.apache.spark.sql.execution.joins.HashJoin
 
buildMetadata(RDD<LabeledPoint>, Strategy, int, String) - Static method in class org.apache.spark.mllib.tree.impl.DecisionTreeMetadata
Construct a DecisionTreeMetadata instance for this dataset and parameters.
buildMetadata(RDD<LabeledPoint>, Strategy) - Static method in class org.apache.spark.mllib.tree.impl.DecisionTreeMetadata
Version of buildMetadata() for DecisionTree.
buildNonNulls() - Method in interface org.apache.spark.sql.columnar.NullableColumnBuilder
 
buildPlan() - Method in interface org.apache.spark.sql.execution.joins.HashJoin
 
buildPools() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
buildPools() - Method in class org.apache.spark.scheduler.FIFOSchedulableBuilder
 
buildPools() - Method in interface org.apache.spark.scheduler.SchedulableBuilder
 
buildProjection() - Method in class org.apache.spark.sql.execution.Project
 
buildRegistryName(Source) - Method in class org.apache.spark.metrics.MetricsSystem
Build a name that uniquely identifies each metric source.
buildScan() - Method in class org.apache.spark.sql.json.JSONRelation
 
buildScan(Seq<Attribute>, Seq<Expression>) - Method in class org.apache.spark.sql.parquet.ParquetRelation2
 
buildScan(Seq<Attribute>, Seq<Expression>) - Method in class org.apache.spark.sql.sources.CatalystScan
 
buildScan(String[], Filter[]) - Method in class org.apache.spark.sql.sources.PrunedFilteredScan
 
buildScan(String[]) - Method in class org.apache.spark.sql.sources.PrunedScan
 
buildScan() - Method in class org.apache.spark.sql.sources.TableScan
 
buildSide() - Method in class org.apache.spark.sql.execution.joins.BroadcastHashJoin
 
buildSide() - Method in class org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoin
 
buildSide() - Method in interface org.apache.spark.sql.execution.joins.HashJoin
 
buildSide() - Method in class org.apache.spark.sql.execution.joins.LeftSemiJoinHash
 
buildSide() - Method in class org.apache.spark.sql.execution.joins.ShuffledHashJoin
 
buildSideKeyGenerator() - Method in interface org.apache.spark.sql.execution.joins.HashJoin
 
BYTE - Class in org.apache.spark.sql.columnar
 
BYTE() - Constructor for class org.apache.spark.sql.columnar.BYTE
 
ByteArrayChunkOutputStream - Class in org.apache.spark.util.io
An OutputStream that writes to fixed-size chunks of byte arrays.
ByteArrayChunkOutputStream(int) - Constructor for class org.apache.spark.util.io.ByteArrayChunkOutputStream
 
ByteArrayColumnType<T extends org.apache.spark.sql.catalyst.types.DataType> - Class in org.apache.spark.sql.columnar
 
ByteArrayColumnType(int, int) - Constructor for class org.apache.spark.sql.columnar.ByteArrayColumnType
 
byteBuffer() - Method in class org.apache.spark.streaming.receiver.ByteBufferBlock
 
ByteBufferBlock - Class in org.apache.spark.streaming.receiver
class representing a block received as an ByteBuffer
ByteBufferBlock(ByteBuffer) - Constructor for class org.apache.spark.streaming.receiver.ByteBufferBlock
 
ByteBufferData - Class in org.apache.spark.streaming.receiver
 
ByteBufferData(ByteBuffer) - Constructor for class org.apache.spark.streaming.receiver.ByteBufferData
 
ByteBufferInputStream - Class in org.apache.spark.util
Reads data from a ByteBuffer, and optionally cleans it up using BlockManager.dispose() at the end of the stream (e.g.
ByteBufferInputStream(ByteBuffer, boolean) - Constructor for class org.apache.spark.util.ByteBufferInputStream
 
ByteBufferValues - Class in org.apache.spark.storage
 
ByteBufferValues(ByteBuffer) - Constructor for class org.apache.spark.storage.ByteBufferValues
 
BytecodeUtils - Class in org.apache.spark.graphx.util
Includes an utility function to test whether a function accesses a specific attribute of an object.
BytecodeUtils() - Constructor for class org.apache.spark.graphx.util.BytecodeUtils
 
ByteColumnAccessor - Class in org.apache.spark.sql.columnar
 
ByteColumnAccessor(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.ByteColumnAccessor
 
ByteColumnBuilder - Class in org.apache.spark.sql.columnar
 
ByteColumnBuilder() - Constructor for class org.apache.spark.sql.columnar.ByteColumnBuilder
 
ByteColumnStats - Class in org.apache.spark.sql.columnar
 
ByteColumnStats() - Constructor for class org.apache.spark.sql.columnar.ByteColumnStats
 
bytes() - Method in class org.apache.spark.streaming.receiver.ByteBufferData
 
BYTES_FOR_PRECISION() - Static method in class org.apache.spark.sql.parquet.ParquetTypesConverter
Compute the FIXED_LEN_BYTE_ARRAY length needed to represent a given DECIMAL precision.
bytesToBytesWritable(byte[]) - Static method in class org.apache.spark.SparkContext
 
bytesToLines(InputStream) - Static method in class org.apache.spark.streaming.dstream.SocketReceiver
This methods translates the data from an inputstream (say, from a socket) to '\n' delimited strings and returns an iterator to access the strings.
bytesToString(long) - Static method in class org.apache.spark.util.Utils
Convert a quantity in bytes to a human-readable string such as "4.0 MB".
bytesWritableConverter() - Static method in class org.apache.spark.SparkContext
 
bytesWritten(long) - Method in interface org.apache.spark.util.logging.RollingPolicy
Notify that bytes have been written
bytesWritten(long) - Method in class org.apache.spark.util.logging.SizeBasedRollingPolicy
Increment the bytes that have been written in the current file
bytesWritten(long) - Method in class org.apache.spark.util.logging.TimeBasedRollingPolicy
 
ByteType - Class in org.apache.spark.sql.api.java
The data type representing byte and Byte values.
ByteType - Static variable in class org.apache.spark.sql.api.java.DataType
Gets the ByteType object.

C

cache() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Persist this RDD with the default storage level (`MEMORY_ONLY`).
cache() - Method in class org.apache.spark.api.java.JavaPairRDD
Persist this RDD with the default storage level (`MEMORY_ONLY`).
cache() - Method in class org.apache.spark.api.java.JavaRDD
Persist this RDD with the default storage level (`MEMORY_ONLY`).
cache() - Method in class org.apache.spark.graphx.Graph
Caches the vertices and edges associated with this graph at the previously-specified target storage levels, which default to MEMORY_ONLY.
cache() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
Persists the edge partitions using `targetStorageLevel`, which defaults to MEMORY_ONLY.
cache() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
cache() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
Persists the vertex partitions at `targetStorageLevel`, which defaults to MEMORY_ONLY.
cache() - Method in class org.apache.spark.partial.StudentTCacher
 
cache() - Method in class org.apache.spark.rdd.RDD
Persist this RDD with the default storage level (`MEMORY_ONLY`).
cache() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
Persist this RDD with the default storage level (`MEMORY_ONLY`).
cache() - Method in class org.apache.spark.sql.SchemaRDD
Overridden cache function will always use the in-memory columnar caching.
cache() - Method in class org.apache.spark.streaming.api.java.JavaDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
cache() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
cache() - Method in class org.apache.spark.streaming.dstream.DStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
CachedBatch - Class in org.apache.spark.sql.columnar
 
CachedBatch(byte[][], Row) - Constructor for class org.apache.spark.sql.columnar.CachedBatch
 
cachedColumnBuffers() - Method in class org.apache.spark.sql.columnar.InMemoryRelation
 
CachedData - Class in org.apache.spark.sql
Holds a cached logical plan and its data
CachedData(LogicalPlan, InMemoryRelation) - Constructor for class org.apache.spark.sql.CachedData
 
cachedData() - Method in interface org.apache.spark.sql.CacheManager
 
cachedRepresentation() - Method in class org.apache.spark.sql.CachedData
 
cacheLock() - Method in interface org.apache.spark.sql.CacheManager
 
CacheManager - Class in org.apache.spark
Spark class responsible for passing RDDs partition contents to the BlockManager and making sure a node doesn't load two copies of an RDD at once.
CacheManager(BlockManager) - Constructor for class org.apache.spark.CacheManager
 
cacheManager() - Method in class org.apache.spark.SparkEnv
 
CacheManager - Interface in org.apache.spark.sql
Provides support in a SQLContext for caching query results and automatically using these cached results when subsequent queries are executed.
cacheQuery(SchemaRDD, Option<String>, StorageLevel) - Method in interface org.apache.spark.sql.CacheManager
Caches the data produced by the logical representation of the given schema rdd.
cacheTable(String) - Method in interface org.apache.spark.sql.CacheManager
Caches the specified table in-memory.
CacheTableCommand - Class in org.apache.spark.sql.execution
:: DeveloperApi ::
CacheTableCommand(String, Option<LogicalPlan>, boolean) - Constructor for class org.apache.spark.sql.execution.CacheTableCommand
 
cacheTables() - Method in class org.apache.spark.sql.hive.test.TestHiveContext
 
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Entropy
:: DeveloperApi :: information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Entropy
:: DeveloperApi :: variance calculation
calculate() - Method in class org.apache.spark.mllib.tree.impurity.EntropyCalculator
Calculate the impurity from the stored sufficient statistics.
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Gini
:: DeveloperApi :: information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Gini
:: DeveloperApi :: variance calculation
calculate() - Method in class org.apache.spark.mllib.tree.impurity.GiniCalculator
Calculate the impurity from the stored sufficient statistics.
calculate(double[], double) - Method in interface org.apache.spark.mllib.tree.impurity.Impurity
:: DeveloperApi :: information calculation for multiclass classification
calculate(double, double, double) - Method in interface org.apache.spark.mllib.tree.impurity.Impurity
:: DeveloperApi :: information calculation for regression
calculate() - Method in class org.apache.spark.mllib.tree.impurity.ImpurityCalculator
Calculate the impurity from the stored sufficient statistics.
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Variance
:: DeveloperApi :: information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Variance
:: DeveloperApi :: variance calculation
calculate() - Method in class org.apache.spark.mllib.tree.impurity.VarianceCalculator
Calculate the impurity from the stored sufficient statistics.
calculatedTasks() - Method in class org.apache.spark.scheduler.TaskSetManager
 
calculateNumBatchesToRemember(Duration) - Static method in class org.apache.spark.streaming.dstream.FileInputDStream
Calculate the number of last batches to remember, such that all the files selected in at least last MIN_REMEMBER_DURATION duration can be remembered.
calculateTotalMemory(SparkContext) - Static method in class org.apache.spark.scheduler.cluster.mesos.MemoryUtils
 
call(T) - Method in interface org.apache.spark.api.java.function.DoubleFlatMapFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.DoubleFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.FlatMapFunction
 
call(T1, T2) - Method in interface org.apache.spark.api.java.function.FlatMapFunction2
 
call(T1) - Method in interface org.apache.spark.api.java.function.Function
 
call(T1, T2) - Method in interface org.apache.spark.api.java.function.Function2
 
call(T1, T2, T3) - Method in interface org.apache.spark.api.java.function.Function3
 
call(T) - Method in interface org.apache.spark.api.java.function.PairFlatMapFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.PairFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.VoidFunction
 
call(T1) - Method in interface org.apache.spark.sql.api.java.UDF1
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10) - Method in interface org.apache.spark.sql.api.java.UDF10
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11) - Method in interface org.apache.spark.sql.api.java.UDF11
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12) - Method in interface org.apache.spark.sql.api.java.UDF12
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13) - Method in interface org.apache.spark.sql.api.java.UDF13
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14) - Method in interface org.apache.spark.sql.api.java.UDF14
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15) - Method in interface org.apache.spark.sql.api.java.UDF15
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16) - Method in interface org.apache.spark.sql.api.java.UDF16
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17) - Method in interface org.apache.spark.sql.api.java.UDF17
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18) - Method in interface org.apache.spark.sql.api.java.UDF18
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19) - Method in interface org.apache.spark.sql.api.java.UDF19
 
call(T1, T2) - Method in interface org.apache.spark.sql.api.java.UDF2
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20) - Method in interface org.apache.spark.sql.api.java.UDF20
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21) - Method in interface org.apache.spark.sql.api.java.UDF21
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21, T22) - Method in interface org.apache.spark.sql.api.java.UDF22
 
call(T1, T2, T3) - Method in interface org.apache.spark.sql.api.java.UDF3
 
call(T1, T2, T3, T4) - Method in interface org.apache.spark.sql.api.java.UDF4
 
call(T1, T2, T3, T4, T5) - Method in interface org.apache.spark.sql.api.java.UDF5
 
call(T1, T2, T3, T4, T5, T6) - Method in interface org.apache.spark.sql.api.java.UDF6
 
call(T1, T2, T3, T4, T5, T6, T7) - Method in interface org.apache.spark.sql.api.java.UDF7
 
call(T1, T2, T3, T4, T5, T6, T7, T8) - Method in interface org.apache.spark.sql.api.java.UDF8
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9) - Method in interface org.apache.spark.sql.api.java.UDF9
 
callSite() - Method in class org.apache.spark.scheduler.ActiveJob
 
callSite() - Method in class org.apache.spark.scheduler.JobSubmitted
 
callSite() - Method in class org.apache.spark.scheduler.Stage
 
CallSite - Class in org.apache.spark.util
CallSite represents a place in user code.
CallSite(String, String) - Constructor for class org.apache.spark.util.CallSite
 
canBeCodeGened(Seq<AggregateExpression>) - Method in class org.apache.spark.sql.execution.SparkStrategies.HashAggregation
 
cancel() - Method in class org.apache.spark.ComplexFutureAction
 
cancel() - Method in interface org.apache.spark.FutureAction
Cancels the execution of this action.
cancel(boolean) - Method in class org.apache.spark.JavaFutureActionWrapper
 
cancel() - Method in class org.apache.spark.scheduler.JobWaiter
Sends a signal to the DAGScheduler to cancel the job.
cancel() - Method in class org.apache.spark.SimpleFutureAction
 
cancel() - Method in class org.apache.spark.util.MetadataCleaner
 
cancelAllJobs() - Method in class org.apache.spark.api.java.JavaSparkContext
Cancel all jobs that have been scheduled or are running.
cancelAllJobs() - Method in class org.apache.spark.scheduler.DAGScheduler
Cancel all jobs that are running or waiting in the queue.
cancelAllJobs() - Method in class org.apache.spark.SparkContext
Cancel all jobs that have been scheduled or are running.
cancelJob(int) - Method in class org.apache.spark.scheduler.DAGScheduler
Cancel a job that is running or waiting in the queue.
cancelJob(int) - Method in class org.apache.spark.SparkContext
Cancel a given job if it's scheduled or running
cancelJobGroup(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Cancel active jobs for the specified group.
cancelJobGroup(String) - Method in class org.apache.spark.scheduler.DAGScheduler
 
cancelJobGroup(String) - Method in class org.apache.spark.SparkContext
Cancel active jobs for the specified group.
cancelStage(int) - Method in class org.apache.spark.scheduler.DAGScheduler
Cancel all jobs associated with a running or scheduled stage.
cancelStage(int) - Method in class org.apache.spark.SparkContext
Cancel a given stage and all jobs associated with it
cancelTasks(int, boolean) - Method in interface org.apache.spark.scheduler.TaskScheduler
 
cancelTasks(int, boolean) - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
canEqual(Object) - Method in class org.apache.spark.sql.api.java.Row
 
canEqual(Object) - Method in class org.apache.spark.util.MutablePair
 
canFetchMoreResults(long) - Method in class org.apache.spark.scheduler.TaskSetManager
Check whether has enough quota to fetch the result with size bytes
capacity() - Method in class org.apache.spark.graphx.impl.VertexPartitionBase
 
cartesian(JavaRDDLike<U, ?>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.
cartesian(RDD<U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.
CartesianPartition - Class in org.apache.spark.rdd
 
CartesianPartition(int, RDD<?>, RDD<?>, int, int) - Constructor for class org.apache.spark.rdd.CartesianPartition
 
CartesianProduct - Class in org.apache.spark.sql.execution.joins
:: DeveloperApi ::
CartesianProduct(SparkPlan, SparkPlan) - Constructor for class org.apache.spark.sql.execution.joins.CartesianProduct
 
CartesianProduct() - Method in class org.apache.spark.sql.execution.SparkStrategies
 
CartesianRDD<T,U> - Class in org.apache.spark.rdd
 
CartesianRDD(SparkContext, RDD<T>, RDD<U>, ClassTag<T>, ClassTag<U>) - Constructor for class org.apache.spark.rdd.CartesianRDD
 
CASE() - Static method in class org.apache.spark.sql.hive.HiveQl
 
caseSensitive() - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog
 
castChildOutput(InsertIntoTable, MetastoreRelation, LogicalPlan) - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog.PreInsertionCasts
 
CatalystArrayContainsNullConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that converts a single-element groups that match the characteristics of an array contains null (see ParquetTypesConverter) into an ArrayType.
CatalystArrayContainsNullConverter(DataType, int, CatalystConverter, Buffer<Object>) - Constructor for class org.apache.spark.sql.parquet.CatalystArrayContainsNullConverter
 
CatalystArrayContainsNullConverter(DataType, int, CatalystConverter) - Constructor for class org.apache.spark.sql.parquet.CatalystArrayContainsNullConverter
 
CatalystArrayConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that converts a single-element groups that match the characteristics of an array (see ParquetTypesConverter) into an ArrayType.
CatalystArrayConverter(DataType, int, CatalystConverter, Buffer<Object>) - Constructor for class org.apache.spark.sql.parquet.CatalystArrayConverter
 
CatalystArrayConverter(DataType, int, CatalystConverter) - Constructor for class org.apache.spark.sql.parquet.CatalystArrayConverter
 
CatalystConverter - Class in org.apache.spark.sql.parquet
 
CatalystConverter() - Constructor for class org.apache.spark.sql.parquet.CatalystConverter
 
CatalystGroupConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that is able to convert a Parquet record to a Row object.
CatalystGroupConverter(StructField[], int, CatalystConverter, ArrayBuffer<Object>, ArrayBuffer<Row>) - Constructor for class org.apache.spark.sql.parquet.CatalystGroupConverter
 
CatalystGroupConverter(StructField[], int, CatalystConverter) - Constructor for class org.apache.spark.sql.parquet.CatalystGroupConverter
 
CatalystGroupConverter(Attribute[]) - Constructor for class org.apache.spark.sql.parquet.CatalystGroupConverter
This constructor is used for the root converter only!
CatalystMapConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that converts two-element groups that match the characteristics of a map (see ParquetTypesConverter) into an MapType.
CatalystMapConverter(StructField[], int, CatalystConverter) - Constructor for class org.apache.spark.sql.parquet.CatalystMapConverter
 
CatalystNativeArrayConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that converts a single-element groups that match the characteristics of an array (see ParquetTypesConverter) into an ArrayType.
CatalystNativeArrayConverter(NativeType, int, CatalystConverter, int) - Constructor for class org.apache.spark.sql.parquet.CatalystNativeArrayConverter
 
CatalystPrimitiveConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.PrimitiveConverter that converts Parquet types to Catalyst types.
CatalystPrimitiveConverter(CatalystConverter, int) - Constructor for class org.apache.spark.sql.parquet.CatalystPrimitiveConverter
 
CatalystPrimitiveRowConverter - Class in org.apache.spark.sql.parquet
A parquet.io.api.GroupConverter that is able to convert a Parquet record to a Row object.
CatalystPrimitiveRowConverter(StructField[], MutableRow) - Constructor for class org.apache.spark.sql.parquet.CatalystPrimitiveRowConverter
 
CatalystPrimitiveRowConverter(Attribute[]) - Constructor for class org.apache.spark.sql.parquet.CatalystPrimitiveRowConverter
 
CatalystScan - Class in org.apache.spark.sql.sources
::Experimental:: An interface for experimenting with a more direct connection to the query planner.
CatalystScan() - Constructor for class org.apache.spark.sql.sources.CatalystScan
 
CatalystStructConverter - Class in org.apache.spark.sql.parquet
This converter is for multi-element groups of primitive or complex types that have repetition level optional or required (so struct fields).
CatalystStructConverter(StructField[], int, CatalystConverter) - Constructor for class org.apache.spark.sql.parquet.CatalystStructConverter
 
Categorical() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
categoricalFeaturesInfo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
categories() - Method in class org.apache.spark.mllib.tree.model.Split
 
category() - Method in class org.apache.spark.mllib.recommendation.ALS.BlockStats
 
category() - Method in class org.apache.spark.mllib.tree.model.Bin
 
channelFactory() - Method in class org.apache.spark.streaming.flume.FlumePollingReceiver
 
channelFactoryExecutor() - Method in class org.apache.spark.streaming.flume.FlumePollingReceiver
 
checkEquals(ASTNode) - Method in class org.apache.spark.sql.hive.HiveQl.TransformableNode
Throws an error if this is not equal to other.
checkHost(String, String) - Static method in class org.apache.spark.util.Utils
 
checkHostPort(String, String) - Static method in class org.apache.spark.util.Utils
 
checkMinimalPollingPeriod(TimeUnit, int) - Static method in class org.apache.spark.metrics.MetricsSystem
 
checkModifyPermissions(String) - Method in class org.apache.spark.SecurityManager
Checks the given user against the modify acl list to see if they have authorization to modify the application.
checkOutputSpecs(JobContext) - Method in class org.apache.spark.sql.parquet.AppendingParquetOutputFormat
 
checkpoint() - Method in interface org.apache.spark.api.java.JavaRDDLike
Mark this RDD for checkpointing.
checkpoint() - Method in class org.apache.spark.graphx.Graph
Mark this Graph for checkpointing.
checkpoint() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
checkpoint() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
checkpoint() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
checkpoint() - Method in class org.apache.spark.rdd.CheckpointRDD
 
checkpoint() - Method in class org.apache.spark.rdd.HadoopRDD
 
checkpoint() - Method in class org.apache.spark.rdd.RDD
Mark this RDD for checkpointing.
checkpoint(Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Enable periodic checkpointing of RDDs of this DStream.
checkpoint(String) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Sets the context to periodically checkpoint the DStream operations for master fault-tolerance.
Checkpoint - Class in org.apache.spark.streaming
 
Checkpoint(StreamingContext, Time) - Constructor for class org.apache.spark.streaming.Checkpoint
 
checkpoint(Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Enable periodic checkpointing of RDDs of this DStream
checkpoint(Duration) - Method in class org.apache.spark.streaming.dstream.ReducedWindowedDStream
 
checkpoint(String) - Method in class org.apache.spark.streaming.StreamingContext
Set the context to periodically checkpoint the DStream operations for driver fault-tolerance.
checkpointBackupFile(String, Time) - Static method in class org.apache.spark.streaming.Checkpoint
Get the checkpoint backup file for the given checkpoint time
checkpointClock() - Method in class org.apache.spark.streaming.kinesis.KinesisCheckpointState
 
checkpointData() - Method in class org.apache.spark.rdd.RDD
 
checkpointData() - Method in class org.apache.spark.streaming.dstream.DStream
 
checkpointDir() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
checkpointDir() - Method in class org.apache.spark.mllib.tree.impl.NodeIdCache
 
checkpointDir() - Method in class org.apache.spark.SparkContext
 
checkpointDir() - Method in class org.apache.spark.streaming.Checkpoint
 
checkpointDir() - Method in class org.apache.spark.streaming.StreamingContext
 
checkpointDirToLogDir(String, int) - Static method in class org.apache.spark.streaming.receiver.WriteAheadLogBasedBlockHandler
 
checkpointDirToLogDir(String) - Static method in class org.apache.spark.streaming.scheduler.ReceivedBlockTracker
 
checkpointDuration() - Method in class org.apache.spark.streaming.Checkpoint
 
checkpointDuration() - Method in class org.apache.spark.streaming.dstream.DStream
 
checkpointDuration() - Method in class org.apache.spark.streaming.StreamingContext
 
Checkpointed() - Static method in class org.apache.spark.rdd.CheckpointState
 
checkpointFile(String, Time) - Static method in class org.apache.spark.streaming.Checkpoint
Get the checkpoint file for the given checkpoint time
CheckpointingInProgress() - Static method in class org.apache.spark.rdd.CheckpointState
 
checkpointInProgress() - Method in class org.apache.spark.streaming.DStreamGraph
 
checkpointInterval() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
checkpointInterval() - Method in class org.apache.spark.mllib.tree.impl.NodeIdCache
 
checkpointPath() - Method in class org.apache.spark.rdd.CheckpointRDD
 
CheckpointRDD<T> - Class in org.apache.spark.rdd
This RDD represents a RDD checkpoint file (similar to HadoopRDD).
CheckpointRDD(SparkContext, String, ClassTag<T>) - Constructor for class org.apache.spark.rdd.CheckpointRDD
 
checkpointRDD() - Method in class org.apache.spark.rdd.RDDCheckpointData
 
CheckpointRDDPartition - Class in org.apache.spark.rdd
 
CheckpointRDDPartition(int) - Constructor for class org.apache.spark.rdd.CheckpointRDDPartition
 
CheckpointReader - Class in org.apache.spark.streaming
 
CheckpointReader() - Constructor for class org.apache.spark.streaming.CheckpointReader
 
CheckpointState - Class in org.apache.spark.rdd
Enumeration to manage state transitions of an RDD through checkpointing [ Initialized --> marked for checkpointing --> checkpointing in progress --> checkpointed ]
CheckpointState() - Constructor for class org.apache.spark.rdd.CheckpointState
 
checkpointTime() - Method in class org.apache.spark.streaming.Checkpoint
 
CheckpointWriter - Class in org.apache.spark.streaming
Convenience class to handle the writing of graph checkpoint to file
CheckpointWriter(JobGenerator, SparkConf, String, Configuration) - Constructor for class org.apache.spark.streaming.CheckpointWriter
 
CheckpointWriter.CheckpointWriteHandler - Class in org.apache.spark.streaming
 
CheckpointWriter.CheckpointWriteHandler(Time, byte[]) - Constructor for class org.apache.spark.streaming.CheckpointWriter.CheckpointWriteHandler
 
checkSpeculatableTasks() - Method in class org.apache.spark.scheduler.Pool
 
checkSpeculatableTasks() - Method in interface org.apache.spark.scheduler.Schedulable
 
checkSpeculatableTasks() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
checkSpeculatableTasks() - Method in class org.apache.spark.scheduler.TaskSetManager
Check for tasks to be speculated and return true if there are any.
checkState(boolean, Function0<String>) - Static method in class org.apache.spark.streaming.util.HdfsUtils
 
checkTimeoutInterval() - Method in class org.apache.spark.storage.BlockManagerMasterActor
 
checkUIViewPermissions(String) - Method in class org.apache.spark.SecurityManager
Checks the given user against the view acl list to see if they have authorization to view the UI.
child() - Method in class org.apache.spark.sql.columnar.InMemoryRelation
 
child() - Method in class org.apache.spark.sql.execution.Aggregate
 
child() - Method in class org.apache.spark.sql.execution.BatchPythonEvaluation
 
child() - Method in class org.apache.spark.sql.execution.DescribeCommand
 
child() - Method in class org.apache.spark.sql.execution.Distinct
 
child() - Method in class org.apache.spark.sql.execution.EvaluatePython
 
child() - Method in class org.apache.spark.sql.execution.Exchange
 
child() - Method in class org.apache.spark.sql.execution.ExternalSort
 
child() - Method in class org.apache.spark.sql.execution.Filter
 
child() - Method in class org.apache.spark.sql.execution.Generate
 
child() - Method in class org.apache.spark.sql.execution.GeneratedAggregate
 
child() - Method in class org.apache.spark.sql.execution.Limit
 
child() - Method in class org.apache.spark.sql.execution.OutputFaker
 
child() - Method in class org.apache.spark.sql.execution.Project
 
child() - Method in class org.apache.spark.sql.execution.Sample
 
child() - Method in class org.apache.spark.sql.execution.Sort
 
child() - Method in class org.apache.spark.sql.execution.TakeOrdered
 
child() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
child() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformation
 
child() - Method in class org.apache.spark.sql.hive.InsertIntoHiveTable
 
child() - Method in class org.apache.spark.sql.parquet.InsertIntoParquetTable
 
children() - Method in class org.apache.spark.sql.columnar.InMemoryRelation
 
children() - Method in class org.apache.spark.sql.execution.BatchPythonEvaluation
 
children() - Method in class org.apache.spark.sql.execution.ExecutedCommand
 
children() - Method in class org.apache.spark.sql.execution.LogicalRDD
 
children() - Method in class org.apache.spark.sql.execution.OutputFaker
 
children() - Method in class org.apache.spark.sql.execution.PythonUDF
 
children() - Method in class org.apache.spark.sql.execution.SparkLogicalPlan
 
children() - Method in class org.apache.spark.sql.execution.Union
 
children() - Method in class org.apache.spark.sql.hive.HiveGenericUdaf
 
children() - Method in class org.apache.spark.sql.hive.HiveGenericUdf
 
children() - Method in class org.apache.spark.sql.hive.HiveGenericUdtf
 
children() - Method in class org.apache.spark.sql.hive.HiveSimpleUdf
 
children() - Method in class org.apache.spark.sql.hive.HiveUdaf
 
children() - Method in class org.apache.spark.sql.hive.InsertIntoHiveTable
 
chiSqFunc() - Method in class org.apache.spark.mllib.stat.test.ChiSqTest.Method
 
chiSqTest(Vector, Vector) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Conduct Pearson's chi-squared goodness of fit test of the observed data against the expected distribution.
chiSqTest(Vector) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Conduct Pearson's chi-squared goodness of fit test of the observed data against the uniform distribution, with each category having an expected frequency of 1 / observed.size.
chiSqTest(Matrix) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Conduct Pearson's independence test on the input contingency matrix, which cannot contain negative entries or columns or rows that sum up to 0.
chiSqTest(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Conduct Pearson's independence test for every feature against the label across the input RDD.
ChiSqTest - Class in org.apache.spark.mllib.stat.test
Conduct the chi-squared test for the input RDDs using the specified method.
ChiSqTest() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest
 
ChiSqTest.Method - Class in org.apache.spark.mllib.stat.test
 
ChiSqTest.Method(String, Function2<Object, Object, Object>) - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.Method
 
ChiSqTest.Method$ - Class in org.apache.spark.mllib.stat.test
 
ChiSqTest.Method$() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.Method$
 
ChiSqTest.NullHypothesis$ - Class in org.apache.spark.mllib.stat.test
 
ChiSqTest.NullHypothesis$() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.NullHypothesis$
 
ChiSqTestResult - Class in org.apache.spark.mllib.stat.test
:: Experimental :: Object containing the test results for the chi-squared hypothesis test.
ChiSqTestResult(double, int, double, String, String) - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
chiSquared(Vector, Vector, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
 
chiSquaredFeatures(RDD<LabeledPoint>, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
Conduct Pearson's independence test for each feature against the label across the input RDD.
chiSquaredMatrix(Matrix, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
 
classForName(String) - Static method in class org.apache.spark.util.Utils
Preferred alternative to Class.forName(className)
Classification() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
ClassificationModel - Interface in org.apache.spark.mllib.classification
:: Experimental :: Represents a classification model that predicts to which of a set of categories an example belongs.
classIsLoadable(String) - Static method in class org.apache.spark.util.Utils
Determines whether the provided class is loadable in the current thread.
classLoader() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
className() - Method in class org.apache.spark.ExceptionFailure
 
classpathEntries() - Method in class org.apache.spark.ui.env.EnvironmentListener
 
classTag() - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
classTag() - Method in class org.apache.spark.api.java.JavaPairRDD
 
classTag() - Method in class org.apache.spark.api.java.JavaRDD
 
classTag() - Method in interface org.apache.spark.api.java.JavaRDDLike
 
classTag() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaDStream
 
classTag() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
clean(F, boolean) - Method in class org.apache.spark.SparkContext
Clean a closure to make it ready to serialized and send to tasks (removes unreferenced variables in $outer's, updates REPL variables) If checkSerializable is set, clean will also proactively check to see if f is serializable and throw a SparkException if not.
clean(Object, boolean) - Static method in class org.apache.spark.util.ClosureCleaner
 
CleanBroadcast - Class in org.apache.spark
 
CleanBroadcast(long) - Constructor for class org.apache.spark.CleanBroadcast
 
cleaner() - Method in class org.apache.spark.SparkContext
 
CleanerListener - Interface in org.apache.spark
Listener class used for testing when any item has been cleaned by the Cleaner class.
CleanRDD - Class in org.apache.spark
 
CleanRDD(int) - Constructor for class org.apache.spark.CleanRDD
 
CleanShuffle - Class in org.apache.spark
 
CleanShuffle(int) - Constructor for class org.apache.spark.CleanShuffle
 
cleanup(long) - Method in class org.apache.spark.SparkContext
Called by MetadataCleaner to clean up the persistentRdds map periodically
cleanup(Time) - Method in class org.apache.spark.streaming.dstream.DStreamCheckpointData
Cleanup old checkpoint data.
cleanup(Time) - Method in class org.apache.spark.streaming.dstream.FileInputDStream.FileInputDStreamCheckpointData
 
cleanUpAfterSchedulerStop() - Method in class org.apache.spark.scheduler.DAGScheduler
 
cleanupOldBatches(Time) - Method in class org.apache.spark.streaming.scheduler.ReceivedBlockTracker
Clean up block information of old batches.
cleanupOldBlock(long) - Method in class org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler
 
cleanupOldBlock(long) - Method in interface org.apache.spark.streaming.receiver.ReceivedBlockHandler
Cleanup old blocks older than the given threshold time
cleanupOldBlock(long) - Method in class org.apache.spark.streaming.receiver.WriteAheadLogBasedBlockHandler
 
cleanupOldLogs(long) - Method in class org.apache.spark.streaming.util.WriteAheadLogManager
Delete the log files that are older than the threshold time.
cleanupOldMetadata(Time) - Method in class org.apache.spark.streaming.scheduler.ReceiverTracker
Clean up metadata older than the given threshold time
CleanupTask - Interface in org.apache.spark
Classes that represent cleaning tasks.
CleanupTaskWeakReference - Class in org.apache.spark
A WeakReference associated with a CleanupTask.
CleanupTaskWeakReference(CleanupTask, Object, ReferenceQueue<Object>) - Constructor for class org.apache.spark.CleanupTaskWeakReference
 
clear() - Static method in class org.apache.spark.Accumulators
 
clear() - Method in interface org.apache.spark.sql.SQLConf
 
clear() - Method in class org.apache.spark.storage.BlockManagerInfo
 
clear() - Method in class org.apache.spark.storage.BlockStore
 
clear() - Method in class org.apache.spark.storage.MemoryStore
 
clear() - Method in class org.apache.spark.util.BoundedPriorityQueue
 
CLEAR_NULL_VALUES_INTERVAL() - Static method in class org.apache.spark.util.TimeStampedWeakValueHashMap
 
clearActiveContext() - Static method in class org.apache.spark.SparkContext
Clears the active SparkContext metadata.
clearCache() - Method in interface org.apache.spark.sql.CacheManager
 
clearCallSite() - Method in class org.apache.spark.api.java.JavaSparkContext
Pass-through to SparkContext.setCallSite.
clearCallSite() - Method in class org.apache.spark.SparkContext
Clear the thread-local property for overriding the call sites of actions and RDDs.
clearCheckpointData(Time) - Method in class org.apache.spark.streaming.dstream.DStream
 
clearCheckpointData(Time) - Method in class org.apache.spark.streaming.DStreamGraph
 
ClearCheckpointData - Class in org.apache.spark.streaming.scheduler
 
ClearCheckpointData(Time) - Constructor for class org.apache.spark.streaming.scheduler.ClearCheckpointData
 
clearDependencies() - Method in class org.apache.spark.rdd.CartesianRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.CoalescedRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.PartitionerAwareUnionRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.ShuffledRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.SubtractedRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.UnionRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.ZippedPartitionsBaseRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD2
 
clearDependencies() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD3
 
clearDependencies() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD4
 
clearFiles() - Method in class org.apache.spark.api.java.JavaSparkContext
Clear the job's list of files added by addFile so that they do not get downloaded to any new nodes.
clearFiles() - Method in class org.apache.spark.SparkContext
Clear the job's list of files added by addFile so that they do not get downloaded to any new nodes.
clearJars() - Method in class org.apache.spark.api.java.JavaSparkContext
Clear the job's list of JARs added by addJar so that they do not get downloaded to any new nodes.
clearJars() - Method in class org.apache.spark.SparkContext
Clear the job's list of JARs added by addJar so that they do not get downloaded to any new nodes.
clearJobGroup() - Method in class org.apache.spark.api.java.JavaSparkContext
Clear the current thread's job group ID and its description.
clearJobGroup() - Method in class org.apache.spark.SparkContext
Clear the current thread's job group ID and its description.
clearMetadata(Time) - Method in class org.apache.spark.streaming.dstream.DStream
Clear metadata that are older than rememberDuration of this DStream.
clearMetadata(Time) - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
Clear metadata that are older than rememberDuration of this DStream.
clearMetadata(Time) - Method in class org.apache.spark.streaming.DStreamGraph
 
ClearMetadata - Class in org.apache.spark.streaming.scheduler
 
ClearMetadata(Time) - Constructor for class org.apache.spark.streaming.scheduler.ClearMetadata
 
clearNullValues() - Method in class org.apache.spark.util.TimeStampedWeakValueHashMap
Remove entries with values that are no longer strongly reachable.
clearOldValues(long, Function2<A, B, BoxedUnit>) - Method in class org.apache.spark.util.TimeStampedHashMap
 
clearOldValues(long) - Method in class org.apache.spark.util.TimeStampedHashMap
Removes old key-value pairs that have timestamp earlier than `threshTime`.
clearOldValues(long) - Method in class org.apache.spark.util.TimeStampedHashSet
Removes old values that have timestamp earlier than threshTime
clearOldValues(long) - Method in class org.apache.spark.util.TimeStampedWeakValueHashMap
Remove old key-value pairs with timestamps earlier than `threshTime`.
clearThreshold() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
:: Experimental :: Clears the threshold so that predict will output raw prediction scores.
clearThreshold() - Method in class org.apache.spark.mllib.classification.SVMModel
:: Experimental :: Clears the threshold so that predict will output raw prediction scores.
client() - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
client() - Method in class org.apache.spark.storage.TachyonBlockManager
 
client() - Method in class org.apache.spark.streaming.flume.FlumeConnection
 
Clock - Interface in org.apache.spark
An abstract clock for measuring elapsed time.
clock() - Method in class org.apache.spark.streaming.scheduler.JobGenerator
 
clock() - Method in class org.apache.spark.streaming.scheduler.JobScheduler
 
Clock - Interface in org.apache.spark.streaming.util
 
Clock - Interface in org.apache.spark.util
An interface to represent clocks, so that they can be mocked out in unit tests.
clone() - Method in class org.apache.spark.mllib.evaluation.binary.BinaryLabelCounter
 
clone() - Method in class org.apache.spark.SparkConf
Copy this object
clone(JvmType) - Method in class org.apache.spark.sql.columnar.ColumnType
Creates a duplicated copy of the value.
clone() - Method in class org.apache.spark.storage.StorageLevel
 
clone() - Method in class org.apache.spark.util.random.BernoulliCellSampler
 
clone() - Method in class org.apache.spark.util.random.BernoulliSampler
 
clone() - Method in class org.apache.spark.util.random.PoissonSampler
 
clone() - Method in interface org.apache.spark.util.random.RandomSampler
return a copy of the RandomSampler object
clone(T, SerializerInstance, ClassTag<T>) - Static method in class org.apache.spark.util.Utils
Clone an object using a Spark serializer.
cloneComplement() - Method in class org.apache.spark.util.random.BernoulliCellSampler
Return a sampler that is the complement of the range specified of the current sampler.
close() - Method in class org.apache.spark.api.java.JavaSparkContext
 
close() - Method in class org.apache.spark.input.FixedLengthBinaryRecordReader
 
close() - Method in class org.apache.spark.input.PortableDataStream
Close the file (if it is currently open)
close() - Method in class org.apache.spark.input.StreamBasedRecordReader
 
close() - Method in class org.apache.spark.input.WholeTextFileRecordReader
 
close() - Method in class org.apache.spark.serializer.DeserializationStream
 
close() - Method in class org.apache.spark.serializer.JavaDeserializationStream
 
close() - Method in class org.apache.spark.serializer.JavaSerializationStream
 
close() - Method in class org.apache.spark.serializer.KryoDeserializationStream
 
close() - Method in class org.apache.spark.serializer.KryoSerializationStream
 
close() - Method in class org.apache.spark.serializer.SerializationStream
 
close() - Method in class org.apache.spark.SparkHadoopWriter
 
close() - Method in class org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer
 
close() - Method in class org.apache.spark.sql.hive.SparkHiveWriterContainer
 
close() - Method in class org.apache.spark.storage.BlockObjectWriter
 
close() - Method in class org.apache.spark.storage.DiskBlockObjectWriter
 
close() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
 
close() - Method in class org.apache.spark.streaming.util.RateLimitedOutputStream
 
close() - Method in class org.apache.spark.streaming.util.WriteAheadLogRandomReader
 
close() - Method in class org.apache.spark.streaming.util.WriteAheadLogReader
 
close() - Method in class org.apache.spark.streaming.util.WriteAheadLogWriter
 
close() - Method in class org.apache.spark.util.FileLogger
Close the writer.
closeIfNeeded() - Method in class org.apache.spark.util.NextIterator
Calls the subclass-defined close method, but only once.
ClosureCleaner - Class in org.apache.spark.util
 
ClosureCleaner() - Constructor for class org.apache.spark.util.ClosureCleaner
 
closureSerializer() - Method in class org.apache.spark.SparkEnv
 
clusterCenters() - Method in class org.apache.spark.mllib.clustering.KMeansModel
 
clusterCenters() - Method in class org.apache.spark.mllib.clustering.StreamingKMeansModel
 
clusterWeights() - Method in class org.apache.spark.mllib.clustering.StreamingKMeansModel
 
cmd() - Method in class org.apache.spark.sql.execution.ExecutedCommand
 
cn() - Method in class org.apache.spark.mllib.feature.VocabWord
 
coalesce(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean, Ordering<Row>) - Method in class org.apache.spark.sql.SchemaRDD
 
CoalescedRDD<T> - Class in org.apache.spark.rdd
Represents a coalesced RDD that has fewer partitions than its parent RDD This class uses the PartitionCoalescer class to find a good partitioning of the parent RDD so that each new partition has roughly the same number of parent partitions and that the preferred location of each new partition overlaps with as many preferred locations of its parent partitions
CoalescedRDD(RDD<T>, int, double, ClassTag<T>) - Constructor for class org.apache.spark.rdd.CoalescedRDD
 
CoalescedRDDPartition - Class in org.apache.spark.rdd
Class that captures a coalesced RDD by essentially keeping track of parent partitions
CoalescedRDDPartition(int, RDD<?>, int[], String) - Constructor for class org.apache.spark.rdd.CoalescedRDDPartition
 
CoarseGrainedClusterMessage - Interface in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages
 
CoarseGrainedClusterMessages.AddWebUIFilter - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.AddWebUIFilter(String, Map<String, String>, String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter
 
CoarseGrainedClusterMessages.AddWebUIFilter$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.AddWebUIFilter$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter$
 
CoarseGrainedClusterMessages.KillExecutors - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutors(Seq<String>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors
 
CoarseGrainedClusterMessages.KillExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors$
 
CoarseGrainedClusterMessages.KillTask - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillTask(long, String, boolean) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
CoarseGrainedClusterMessages.KillTask$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillTask$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask$
 
CoarseGrainedClusterMessages.LaunchTask - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.LaunchTask(SerializableBuffer) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask
 
CoarseGrainedClusterMessages.LaunchTask$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.LaunchTask$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask$
 
CoarseGrainedClusterMessages.RegisterClusterManager$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterClusterManager$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterClusterManager$
 
CoarseGrainedClusterMessages.RegisteredExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisteredExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisteredExecutor$
 
CoarseGrainedClusterMessages.RegisterExecutor - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutor(String, String, int) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
CoarseGrainedClusterMessages.RegisterExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor$
 
CoarseGrainedClusterMessages.RegisterExecutorFailed - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutorFailed(String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed
 
CoarseGrainedClusterMessages.RegisterExecutorFailed$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutorFailed$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed$
 
CoarseGrainedClusterMessages.RemoveExecutor - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveExecutor(String, String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
 
CoarseGrainedClusterMessages.RemoveExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor$
 
CoarseGrainedClusterMessages.RequestExecutors - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RequestExecutors(int) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
CoarseGrainedClusterMessages.RequestExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RequestExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors$
 
CoarseGrainedClusterMessages.RetrieveSparkProps$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RetrieveSparkProps$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveSparkProps$
 
CoarseGrainedClusterMessages.ReviveOffers$ - Class in org.apache.spark.scheduler.cluster
Alternate factory method that takes a ByteBuffer directly for the data field
CoarseGrainedClusterMessages.ReviveOffers$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.ReviveOffers$
 
CoarseGrainedClusterMessages.StatusUpdate - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StatusUpdate(String, long, Enumeration.Value, SerializableBuffer) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
CoarseGrainedClusterMessages.StatusUpdate$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StatusUpdate$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate$
 
CoarseGrainedClusterMessages.StopDriver$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopDriver$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopDriver$
 
CoarseGrainedClusterMessages.StopExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutor$
 
CoarseGrainedClusterMessages.StopExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutors$
 
CoarseGrainedSchedulerBackend - Class in org.apache.spark.scheduler.cluster
A scheduler backend that waits for coarse grained executors to connect to it through Akka.
CoarseGrainedSchedulerBackend(TaskSchedulerImpl, ActorSystem) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
CoarseGrainedSchedulerBackend.DriverActor - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedSchedulerBackend.DriverActor(Seq<Tuple2<String, String>>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.DriverActor
 
CoarseMesosSchedulerBackend - Class in org.apache.spark.scheduler.cluster.mesos
A SchedulerBackend that runs tasks on Mesos, but uses "coarse-grained" tasks, where it holds onto each Mesos node for the duration of the Spark job instead of relinquishing cores whenever a task is done.
CoarseMesosSchedulerBackend(TaskSchedulerImpl, SparkContext, String) - Constructor for class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
code() - Method in class org.apache.spark.mllib.feature.VocabWord
 
codegenEnabled() - Method in class org.apache.spark.sql.execution.SparkPlan
 
codegenEnabled() - Method in interface org.apache.spark.sql.SQLConf
When set to true, Spark SQL will use the Scala compiler at runtime to generate custom bytecode that evaluates expressions found in queries.
codeLen() - Method in class org.apache.spark.mllib.feature.VocabWord
 
cogroup(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
 
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
 
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
 
cogroup(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
CoGroupedRDD<K> - Class in org.apache.spark.rdd
:: DeveloperApi :: A RDD that cogroups its parents.
CoGroupedRDD(Seq<RDD<? extends Product2<K, ?>>>, Partitioner) - Constructor for class org.apache.spark.rdd.CoGroupedRDD
 
CoGroupPartition - Class in org.apache.spark.rdd
 
CoGroupPartition(int, CoGroupSplitDep[]) - Constructor for class org.apache.spark.rdd.CoGroupPartition
 
cogroupResult2ToJava(RDD<Tuple2<K, Tuple3<Iterable<V>, Iterable<W1>, Iterable<W2>>>>, ClassTag<K>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
cogroupResult3ToJava(RDD<Tuple2<K, Tuple4<Iterable<V>, Iterable<W1>, Iterable<W2>, Iterable<W3>>>>, ClassTag<K>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
cogroupResultToJava(RDD<Tuple2<K, Tuple2<Iterable<V>, Iterable<W>>>>, ClassTag<K>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
CoGroupSplitDep - Interface in org.apache.spark.rdd
 
collect() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an array that contains all of the elements in this RDD.
collect() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
collect() - Method in class org.apache.spark.rdd.RDD
Return an array that contains all of the elements in this RDD.
collect(PartialFunction<T, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return an RDD that contains all matching values by applying f.
collect() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
 
collect() - Method in class org.apache.spark.sql.SchemaRDD
 
collectAsMap() - Method in class org.apache.spark.api.java.JavaPairRDD
Return the key-value pairs in this RDD to the master as a Map.
collectAsMap() - Method in class org.apache.spark.rdd.PairRDDFunctions
Return the key-value pairs in this RDD to the master as a Map.
collectAsync() - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of collect, which returns a future for retrieving an array containing all of the elements in this RDD.
collectAsync() - Method in class org.apache.spark.rdd.AsyncRDDActions
Returns a future for retrieving all elements of this RDD.
collectEdges(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Returns an RDD that contains for each vertex v its local edges, i.e., the edges that are incident on v, in the user-specified direction.
collectedStatistics() - Method in class org.apache.spark.sql.columnar.BinaryColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.BooleanColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.ByteColumnStats
 
collectedStatistics() - Method in interface org.apache.spark.sql.columnar.ColumnStats
Column statistics represented as a single row, currently including closed lower bound, closed upper bound and null count.
collectedStatistics() - Method in class org.apache.spark.sql.columnar.DateColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.DoubleColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.FloatColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.GenericColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.IntColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.LongColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.NoopColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.ShortColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.StringColumnStats
 
collectedStatistics() - Method in class org.apache.spark.sql.columnar.TimestampColumnStats
 
CollectionsUtils - Class in org.apache.spark.util
 
CollectionsUtils() - Constructor for class org.apache.spark.util.CollectionsUtils
 
collectNeighborIds(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Collect the neighbor vertex ids for each vertex.
collectNeighbors(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Collect the neighbor vertex attributes for each vertex.
collectPartitions(int[]) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an array that contains all of the elements in a specific partition of this RDD.
collectPartitions() - Method in class org.apache.spark.rdd.RDD
A private method for tests, to look at the contents of each partition
collectToPython() - Method in class org.apache.spark.sql.SchemaRDD
Serializes the Array[Row] returned by SchemaRDD's optimized collect(), using the same format as javaToPython.
colPtrs() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
colStats(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Computes column-wise summary statistics for the input RDD[Vector].
ColumnAccessor - Interface in org.apache.spark.sql.columnar
An Iterator like trait used to extract values from columnar byte buffer.
columnBatchSize() - Method in interface org.apache.spark.sql.SQLConf
The number of rows that will be
ColumnBuilder - Interface in org.apache.spark.sql.columnar
 
columnNameOfCorruptRecord() - Method in interface org.apache.spark.sql.SQLConf
 
columnOrdinals() - Method in class org.apache.spark.sql.hive.MetastoreRelation
An attribute map for determining the ordinal for non-partition columns.
columnPruningPred() - Method in class org.apache.spark.sql.parquet.ParquetTableScan
 
columnSimilarities() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Compute all cosine similarities between columns of this matrix using the brute-force approach of computing normalized dot products.
columnSimilarities(double) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Compute similarities between columns of this matrix using a sampling approach.
columnSimilaritiesDIMSUM(double[], double) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Find all similar columns using the DIMSUM sampling algorithm, described in two papers
ColumnStatisticsSchema - Class in org.apache.spark.sql.columnar
 
ColumnStatisticsSchema(Attribute) - Constructor for class org.apache.spark.sql.columnar.ColumnStatisticsSchema
 
columnStats() - Method in class org.apache.spark.sql.columnar.BasicColumnBuilder
 
columnStats() - Method in interface org.apache.spark.sql.columnar.ColumnBuilder
Column statistics information
ColumnStats - Interface in org.apache.spark.sql.columnar
Used to collect statistical information when building in-memory columns.
columnStats() - Method in class org.apache.spark.sql.columnar.NativeColumnBuilder
 
columnType() - Method in class org.apache.spark.sql.columnar.BasicColumnBuilder
 
ColumnType<T extends org.apache.spark.sql.catalyst.types.DataType,JvmType> - Class in org.apache.spark.sql.columnar
An abstract class that represents type of a column.
ColumnType(int, int) - Constructor for class org.apache.spark.sql.columnar.ColumnType
 
columnType() - Method in class org.apache.spark.sql.columnar.NativeColumnBuilder
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Simplified version of combineByKey that hash-partitions the output RDD.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Method in class org.apache.spark.api.java.JavaPairRDD
Simplified version of combineByKey that hash-partitions the resulting RDD using the existing partitioner/parallelism level.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, Serializer) - Method in class org.apache.spark.rdd.PairRDDFunctions
Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Simplified version of combineByKey that hash-partitions the output RDD.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Method in class org.apache.spark.rdd.PairRDDFunctions
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Combine elements of each key in DStream's RDDs using custom function.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Combine elements of each key in DStream's RDDs using custom function.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, ClassTag<C>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Combine elements of each key in DStream's RDDs using custom functions.
combineCombinersByKey(Iterator<Product2<K, C>>) - Method in class org.apache.spark.Aggregator
 
combineCombinersByKey(Iterator<Product2<K, C>>, TaskContext) - Method in class org.apache.spark.Aggregator
 
combineValuesByKey(Iterator<Product2<K, V>>) - Method in class org.apache.spark.Aggregator
 
combineValuesByKey(Iterator<Product2<K, V>>, TaskContext) - Method in class org.apache.spark.Aggregator
 
Command - Interface in org.apache.spark.sql.execution
 
command() - Method in class org.apache.spark.sql.execution.PythonUDF
 
commands() - Method in class org.apache.spark.sql.hive.test.TestHiveContext.TestTable
 
commit() - Method in class org.apache.spark.SparkHadoopWriter
 
commitAndClose() - Method in class org.apache.spark.storage.BlockObjectWriter
Flush the partial writes and commit them as a single atomic block.
commitAndClose() - Method in class org.apache.spark.storage.DiskBlockObjectWriter
 
commitJob() - Method in class org.apache.spark.SparkHadoopWriter
 
commitJob() - Method in class org.apache.spark.sql.hive.SparkHiveDynamicPartitionWriterContainer
 
commitJob() - Method in class org.apache.spark.sql.hive.SparkHiveWriterContainer
 
commonHeaderNodes() - Static method in class org.apache.spark.ui.UIUtils
 
comparator(Schedulable, Schedulable) - Method in class org.apache.spark.scheduler.FairSchedulingAlgorithm
 
comparator(Schedulable, Schedulable) - Method in class org.apache.spark.scheduler.FIFOSchedulingAlgorithm
 
comparator(Schedulable, Schedulable) - Method in interface org.apache.spark.scheduler.SchedulingAlgorithm
 
compare(PartitionGroup, PartitionGroup) - Method in class org.apache.spark.rdd.PartitionCoalescer
 
compare(Option<PartitionGroup>, Option<PartitionGroup>) - Method in class org.apache.spark.rdd.PartitionCoalescer
 
compare(RDDInfo) - Method in class org.apache.spark.storage.RDDInfo
 
compatibilityBlackList() - Static method in class org.apache.spark.sql.hive.HiveShim
 
compatibleType(DataType, DataType) - Static method in class org.apache.spark.sql.json.JsonRDD
Returns the most general data type for two given data types.
completedIndices() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
completedJobs() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
completedStageIndices() - Method in class org.apache.spark.ui.jobs.UIData.JobUIData
 
completedStages() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
completedTasks() - Method in class org.apache.spark.ui.exec.ExecutorSummaryInfo
 
completion() - Method in class org.apache.spark.util.CompletionIterator
 
CompletionEvent - Class in org.apache.spark.scheduler
 
CompletionEvent(Task<?>, TaskEndReason, Object, Map<Object, Object>, TaskInfo, TaskMetrics) - Constructor for class org.apache.spark.scheduler.CompletionEvent
 
CompletionIterator<A,I extends scala.collection.Iterator<A>> - Class in org.apache.spark.util
Wrapper around an iterator which calls a completion method after it successfully iterates through all the elements.
CompletionIterator(I) - Constructor for class org.apache.spark.util.CompletionIterator
 
completionTime() - Method in class org.apache.spark.scheduler.StageInfo
Time when all tasks in the stage completed or when the stage was cancelled.
ComplexColumnBuilder<T extends org.apache.spark.sql.catalyst.types.DataType,JvmType> - Class in org.apache.spark.sql.columnar
 
ComplexColumnBuilder(ColumnStats, ColumnType<T, JvmType>) - Constructor for class org.apache.spark.sql.columnar.ComplexColumnBuilder
 
ComplexFutureAction<T> - Class in org.apache.spark
A FutureAction for actions that could trigger multiple Spark jobs.
ComplexFutureAction() - Constructor for class org.apache.spark.ComplexFutureAction
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.BooleanBitSet.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.DictionaryEncoding.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in interface org.apache.spark.sql.columnar.compression.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.IntDelta.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.LongDelta.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.PassThrough.Encoder
 
compress(ByteBuffer, ByteBuffer) - Method in class org.apache.spark.sql.columnar.compression.RunLengthEncoding.Encoder
 
compressCodec() - Method in class org.apache.spark.sql.hive.ShimFileSinkDesc
 
compressed() - Method in class org.apache.spark.sql.hive.ShimFileSinkDesc
 
compressedInputStream(InputStream) - Method in interface org.apache.spark.io.CompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.LZ4CompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.LZFCompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.SnappyCompressionCodec
 
CompressedMapStatus - Class in org.apache.spark.scheduler
A MapStatus implementation that tracks the size of each block.
CompressedMapStatus(BlockManagerId, byte[]) - Constructor for class org.apache.spark.scheduler.CompressedMapStatus
 
CompressedMapStatus(BlockManagerId, long[]) - Constructor for class org.apache.spark.scheduler.CompressedMapStatus
 
compressedOutputStream(OutputStream) - Method in interface org.apache.spark.io.CompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.LZ4CompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.LZFCompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.SnappyCompressionCodec
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.BooleanBitSet.Encoder
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.DictionaryEncoding.Encoder
 
compressedSize() - Method in interface org.apache.spark.sql.columnar.compression.Encoder
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.IntDelta.Encoder
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.LongDelta.Encoder
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.PassThrough.Encoder
 
compressedSize() - Method in class org.apache.spark.sql.columnar.compression.RunLengthEncoding.Encoder
 
CompressibleColumnAccessor<T extends org.apache.spark.sql.catalyst.types.NativeType> - Interface in org.apache.spark.sql.columnar.compression
 
CompressibleColumnBuilder<T extends org.apache.spark.sql.catalyst.types.NativeType> - Interface in org.apache.spark.sql.columnar.compression
A stackable trait that builds optionally compressed byte buffer for a column.
COMPRESSION_CODEC_PREFIX() - Static method in class org.apache.spark.scheduler.EventLoggingListener
 
CompressionCodec - Interface in org.apache.spark.io
:: DeveloperApi :: CompressionCodec allows the customization of choosing different compression implementations to be used in block storage.
compressionCodec() - Method in class org.apache.spark.scheduler.EventLoggingInfo
 
compressionCodec() - Method in class org.apache.spark.streaming.CheckpointWriter
 
compressionEncoders() - Method in interface org.apache.spark.sql.columnar.compression.CompressibleColumnBuilder
 
compressionRatio() - Method in interface org.apache.spark.sql.columnar.compression.Encoder
 
CompressionScheme - Interface in org.apache.spark.sql.columnar.compression
 
compressType() - Method in class org.apache.spark.sql.hive.ShimFileSinkDesc
 
compute(Partition, TaskContext) - Method in class org.apache.spark.graphx.EdgeRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.graphx.VertexRDD
Provides the RDD[(VertexId, VD)] equivalent output.
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.Gradient
Compute the gradient and loss given the features of a single data point.
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.Gradient
Compute the gradient and loss given the features of a single data point, add the gradient to a provided vector to avoid creating new objects, and return loss.
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.HingeGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.HingeGradient
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.L1Updater
 
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.LeastSquaresGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.LeastSquaresGradient
 
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.LogisticGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.LogisticGradient
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.SimpleUpdater
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.SquaredL2Updater
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.Updater
Compute an updated value for weights given the gradient, stepSize, iteration number and regularization parameter.
compute(Partition, TaskContext) - Method in class org.apache.spark.mllib.rdd.RandomRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.mllib.rdd.RandomVectorRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.mllib.rdd.SlidingRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.BlockRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.CartesianRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.CheckpointRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.CoalescedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.CoGroupedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.EmptyRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.FilteredRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.FlatMappedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.FlatMappedValuesRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.GlommedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.HadoopRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.HadoopRDD.HadoopMapPartitionsWithSplitRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.JdbcRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.MapPartitionsRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.MappedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.MappedValuesRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.NewHadoopRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.NewHadoopRDD.NewHadoopMapPartitionsWithSplitRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ParallelCollectionRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.PartitionerAwareUnionRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.PartitionPruningRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.PartitionwiseSampledRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.PipedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.RDD
:: DeveloperApi :: Implemented by subclasses to compute a given partition.
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.SampledRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ShuffledRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.SubtractedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.UnionRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ZippedPartitionsRDD2
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ZippedPartitionsRDD3
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ZippedPartitionsRDD4
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ZippedWithIndexRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.sql.SchemaRDD
 
compute(Time) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Generate an RDD for the given duration
compute(Time) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Method that generates a RDD for the given Duration
compute(Time) - Method in class org.apache.spark.streaming.dstream.ConstantInputDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.DStream
Method that generates a RDD for the given time
compute(Time) - Method in class org.apache.spark.streaming.dstream.FileInputDStream
Finds the files that were modified since the last time this method was called and makes a union RDD out of them.
compute(Time) - Method in class org.apache.spark.streaming.dstream.FilteredDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.FlatMappedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.FlatMapValuedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.ForEachDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.GlommedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.MapPartitionedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.MappedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.MapValuedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.QueueInputDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
Generates RDDs with blocks received by the receiver of this stream.
compute(Time) - Method in class org.apache.spark.streaming.dstream.ReducedWindowedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.ShuffledDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.StateDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.TransformedDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.UnionDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.WindowedDStream
 
compute(Partition, TaskContext) - Method in class org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD
Gets the partition data by getting the corresponding block from the block manager.
computeColumnSummaryStatistics() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes column-wise summary statistics.
computeCorrelation(RDD<Object>, RDD<Object>) - Method in interface org.apache.spark.mllib.stat.correlation.Correlation
Compute correlation for two datasets.
computeCorrelation(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation for two datasets.
computeCorrelation(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
Compute Spearman's correlation for two datasets.
computeCorrelationMatrix(RDD<Vector>) - Method in interface org.apache.spark.mllib.stat.correlation.Correlation
Compute the correlation matrix S, for the input matrix, where S(i, j) is the correlation between column i and j.
computeCorrelationMatrix(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation matrix S, for the input matrix, where S(i, j) is the correlation between column i and j.
computeCorrelationMatrix(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
Compute Spearman's correlation matrix S, for the input matrix, where S(i, j) is the correlation between column i and j.
computeCorrelationMatrixFromCovariance(Matrix) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation matrix from the covariance matrix.
computeCorrelationWithMatrixImpl(RDD<Object>, RDD<Object>) - Method in interface org.apache.spark.mllib.stat.correlation.Correlation
Combine the two input RDD[Double]s into an RDD[Vector] and compute the correlation using the correlation implementation for RDD[Vector].
computeCost(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.KMeansModel
Return the K-means cost (sum of squared distances of points to their nearest center) for this model on the given data.
computeCovariance() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the covariance matrix, treating each row as an observation.
computeError(TreeEnsembleModel, RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.tree.loss.AbsoluteError
Method to calculate loss of the base learner for the gradient boosting calculation.
computeError(TreeEnsembleModel, RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.tree.loss.LogLoss
Method to calculate loss of the base learner for the gradient boosting calculation.
computeError(TreeEnsembleModel, RDD<LabeledPoint>) - Method in interface org.apache.spark.mllib.tree.loss.Loss
Method to calculate error of the base learner for the gradient boosting calculation.
computeError(TreeEnsembleModel, RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.tree.loss.SquaredError
Method to calculate loss of the base learner for the gradient boosting calculation.
computeFractionForSampleSize(int, long, boolean) - Static method in class org.apache.spark.util.random.SamplingUtils
Returns a sampling rate that guarantees a sample of size >= sampleSizeLowerBound 99.99% of the time.
computeGramianMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Computes the Gramian matrix A^T A.
computeGramianMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the Gramian matrix A^T A.
computeOrReadCheckpoint(Partition, TaskContext) - Method in class org.apache.spark.rdd.RDD
Compute an RDD partition or read it from a checkpoint if the RDD is checkpointing.
computePreferredLocations(Seq<InputFormatInfo>) - Static method in class org.apache.spark.scheduler.InputFormatInfo
Computes the preferred locations based on input(s) and returned a location to block map.
computePrincipalComponents(int) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the top k principal components.
computeSplitSize(long, long, long) - Method in class org.apache.spark.input.FixedLengthBinaryInputFormat
This input format overrides computeSplitSize() to make sure that each split only contains full records.
computeSVD(int, boolean, double) - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Computes the singular value decomposition of this IndexedRowMatrix.
computeSVD(int, boolean, double) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes singular value decomposition of this matrix.
computeSVD(int, boolean, double, int, double, String) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
The actual SVD implementation, visible for testing.
computeThresholdByKey(Map<K, AcceptanceResult>, Map<K, Object>) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Given the result returned by getCounts, determine the threshold for accepting items to generate exact sample size.
condition() - Method in class org.apache.spark.sql.execution.Filter
 
condition() - Method in class org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoin
 
condition() - Method in class org.apache.spark.sql.execution.joins.HashOuterJoin
 
condition() - Method in class org.apache.spark.sql.execution.joins.LeftSemiJoinBNL
 
conditionEvaluator() - Method in class org.apache.spark.sql.execution.Filter
 
conf() - Method in class org.apache.spark.rdd.RDD
 
conf() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
conf() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
conf() - Method in class org.apache.spark.scheduler.TaskSetManager
 
conf() - Method in class org.apache.spark.SparkContext
 
conf() - Method in class org.apache.spark.SparkEnv
 
conf() - Method in class org.apache.spark.sql.parquet.ParquetRelation
 
conf() - Method in class org.apache.spark.storage.BlockManager
 
conf() - Method in class org.apache.spark.streaming.StreamingContext
 
conf() - Method in class org.apache.spark.ui.SparkUI
 
confidence() - Method in class org.apache.spark.partial.BoundedDouble
 
configFile() - Method in class org.apache.spark.metrics.MetricsConfig
 
configTestLog4j(String) - Static method in class org.apache.spark.util.Utils
config a log4j properties used for testsuite
configuration() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
CONFIGURATION_INSTANTIATION_LOCK() - Static method in class org.apache.spark.rdd.HadoopRDD
Configuration's constructor is not threadsafe (see SPARK-1097 and HADOOP-10456).
confusionMatrix() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns confusion matrix: predicted classes are in columns, they are ordered by class label ascending, as in "labels"
connected(String) - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
connectedComponents() - Method in class org.apache.spark.graphx.GraphOps
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
ConnectedComponents - Class in org.apache.spark.graphx.lib
Connected components algorithm.
ConnectedComponents() - Constructor for class org.apache.spark.graphx.lib.ConnectedComponents
 
CONSOLE_DEFAULT_PERIOD() - Method in class org.apache.spark.metrics.sink.ConsoleSink
 
CONSOLE_DEFAULT_UNIT() - Method in class org.apache.spark.metrics.sink.ConsoleSink
 
CONSOLE_KEY_PERIOD() - Method in class org.apache.spark.metrics.sink.ConsoleSink
 
CONSOLE_KEY_UNIT() - Method in class org.apache.spark.metrics.sink.ConsoleSink
 
ConsoleProgressBar - Class in org.apache.spark.ui
ConsoleProgressBar shows the progress of stages in the next line of the console.
ConsoleProgressBar(SparkContext) - Constructor for class org.apache.spark.ui.ConsoleProgressBar
 
ConsoleSink - Class in org.apache.spark.metrics.sink
 
ConsoleSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.spark.metrics.sink.ConsoleSink
 
ConstantInputDStream<T> - Class in org.apache.spark.streaming.dstream
An input stream that always returns the same RDD on each timestep.
ConstantInputDStream(StreamingContext, RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.ConstantInputDStream
 
constructURIForAuthentication(URI, SecurityManager) - Static method in class org.apache.spark.util.Utils
Construct a URI container information used for authentication.
consumerConnector() - Method in class org.apache.spark.streaming.kafka.KafkaReceiver
 
contains(Param<?>) - Method in class org.apache.spark.ml.param.ParamMap
Checks whether a parameter is explicitly specified.
contains(String) - Method in class org.apache.spark.SparkConf
Does the configuration contain a given parameter?
contains(BlockId) - Method in class org.apache.spark.storage.BlockManagerMaster
Check if block manager master has a block.
contains(BlockId) - Method in class org.apache.spark.storage.BlockStore
 
contains(BlockId) - Method in class org.apache.spark.storage.DiskStore
 
contains(BlockId) - Method in class org.apache.spark.storage.MemoryStore
 
contains(BlockId) - Method in class org.apache.spark.storage.TachyonStore
 
contains(A) - Method in class org.apache.spark.util.TimeStampedHashSet
 
containsBlock(BlockId) - Method in class org.apache.spark.storage.DiskBlockManager
Check if disk block manager has a block.
containsBlock(BlockId) - Method in class org.apache.spark.storage.StorageStatus
Return whether the given block is stored in this block manager in O(1) time.
containsCachedMetadata(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
containsShuffle(int) - Method in class org.apache.spark.MapOutputTrackerMaster
Check if the given shuffle is being tracked
contentType() - Method in class org.apache.spark.ui.JettyUtils.ServletParams
 
context() - Method in interface org.apache.spark.api.java.JavaRDDLike
The SparkContext that this RDD was created on.
context() - Method in class org.apache.spark.InterruptibleIterator
 
context() - Method in class org.apache.spark.rdd.RDD
The SparkContext that this RDD was created on.
context() - Method in class org.apache.spark.sql.execution.SparkStrategies.CommandStrategy
 
context() - Method in class org.apache.spark.sql.hive.execution.HiveTableScan
 
context() - Method in class org.apache.spark.sql.hive.HiveStrategies.HiveCommandStrategy
 
context() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return the StreamingContext associated with this DStream
context() - Method in class org.apache.spark.streaming.dstream.DStream
Return the StreamingContext associated with this DStream
ContextCleaner - Class in org.apache.spark
An asynchronous cleaner for RDD, shuffle, and broadcast state.
ContextCleaner(SparkContext) - Constructor for class org.apache.spark.ContextCleaner
 
ContextWaiter - Class in org.apache.spark.streaming
 
ContextWaiter() - Constructor for class org.apache.spark.streaming.ContextWaiter
 
Continuous() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
convert() - Method in class org.apache.spark.WritableConverter
 
convertCatalystToJava(Object) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Converts Java objects to catalyst rows / types
convertFromAttributes(Seq<Attribute>) - Static method in class org.apache.spark.sql.parquet.ParquetTypesConverter
 
convertFromString(String) - Static method in class org.apache.spark.sql.parquet.ParquetTypesConverter
 
convertJavaToCatalyst(Object, DataType) - Static method in class org.apache.spark.sql.types.util.DataTypeConversions
Converts Java objects to catalyst rows / types
convertMetastoreParquet() - Method in class org.apache.spark.sql.hive.HiveContext
When true, enables an experimental feature where metastore tables that use the parquet SerDe are automatically converted to use the Spark SQL parquet table scan, instead of the Hive SerDe.
convertSplitLocationInfo(Object[]) - Static method in class org.apache.spark.rdd.HadoopRDD
 
convertToAttributes(Type, boolean) - Static method in class org.apache.spark.sql.parquet.ParquetTypesConverter
 
convertToBaggedRDD(RDD<Datum>, double, int, boolean, int) - Static method in class org.apache.spark.mllib.tree.impl.BaggedPoint
Convert an input dataset into its BaggedPoint representation, choosing subsamplingRate counts for each instance.
convertToString(Seq<Attribute>) - Static method in class org.apache.spark.sql.parquet.ParquetTypesConverter
 
convertToTreeRDD(RDD<LabeledPoint>, Bin[][], DecisionTreeMetadata) - Static method in class org.apache.spark.mllib.tree.impl.TreePoint
Convert an input dataset into its TreePoint representation, binning feature values in preparation for DecisionTree training.
CoordinateMatrix - Class in org.apache.spark.mllib.linalg.distributed
:: Experimental :: Represents a matrix in coordinate format.
CoordinateMatrix(RDD<MatrixEntry>, long, long) - Constructor for class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
 
CoordinateMatrix(RDD<MatrixEntry>) - Constructor for class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Alternative constructor leaving matrix dimensions to be determined automatically.
copiesRunning() - Method in class org.apache.spark.scheduler.TaskSetManager
 
copy() - Method in class org.apache.spark.ml.param.ParamMap
Make a copy of this param map.
copy(Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
y = x
copy() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
copy() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
copy() - Method in interface org.apache.spark.mllib.linalg.Matrix
Get a deep copy of the matrix.
copy() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
copy() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
copy() - Method in interface org.apache.spark.mllib.linalg.Vector
Makes a deep copy of this vector.
copy() - Method in class org.apache.spark.mllib.random.PoissonGenerator
 
copy() - Method in interface org.apache.spark.mllib.random.RandomDataGenerator
Returns a copy of the RandomDataGenerator with a new instance of the rng object used in the class when applicable for non-locking concurrent usage.
copy() - Method in class org.apache.spark.mllib.random.StandardNormalGenerator
 
copy() - Method in class org.apache.spark.mllib.random.UniformGenerator
 
copy() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
Returns a shallow copy of this instance.
copy() - Method in class org.apache.spark.mllib.tree.impurity.EntropyCalculator
Make a deep copy of this ImpurityCalculator.
copy() - Method in class org.apache.spark.mllib.tree.impurity.GiniCalculator
Make a deep copy of this ImpurityCalculator.
copy() - Method in class org.apache.spark.mllib.tree.impurity.ImpurityCalculator
Make a deep copy of this ImpurityCalculator.
copy() - Method in class org.apache.spark.mllib.tree.impurity.VarianceCalculator
Make a deep copy of this ImpurityCalculator.
copy() - Method in class org.apache.spark.util.StatCounter
Clone this StatCounter
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.BOOLEAN
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.BYTE
 
copyField(Row, int, MutableRow, int) - Method in class org.apache.spark.sql.columnar.ColumnType
Copies from(fromOrdinal) to to(toOrdinal).
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.DOUBLE
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.FLOAT
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.INT
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.LONG
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.SHORT
 
copyField(Row, int, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.STRING
 
copyStream(InputStream, OutputStream, boolean, boolean) - Static method in class org.apache.spark.util.Utils
Copy all data from an InputStream to an OutputStream.
cores() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
cores() - Method in class org.apache.spark.scheduler.WorkerOffer
 
coresByTaskId() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
corr(RDD<Object>, RDD<Object>, String) - Static method in class org.apache.spark.mllib.stat.correlation.Correlations
 
corr(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Compute the Pearson correlation matrix for the input RDD of Vectors.
corr(RDD<Vector>, String) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Compute the correlation matrix for the input RDD of Vectors using the specified method.
corr(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Compute the Pearson correlation for the input RDDs.
corr(RDD<Object>, RDD<Object>, String) - Static method in class org.apache.spark.mllib.stat.Statistics
:: Experimental :: Compute the correlation for the input RDDs using the specified method.
Correlation - Interface in org.apache.spark.mllib.stat.correlation
Trait for correlation algorithms.
CorrelationNames - Class in org.apache.spark.mllib.stat.correlation
Maintains supported and default correlation names.
CorrelationNames() - Constructor for class org.apache.spark.mllib.stat.correlation.CorrelationNames
 
Correlations - Class in org.apache.spark.mllib.stat.correlation
Delegates computation to the specific correlation object based on the input method name.
Correlations() - Constructor for class org.apache.spark.mllib.stat.correlation.Correlations
 
corrMatrix(RDD<Vector>, String) - Static method in class org.apache.spark.mllib.stat.correlation.Correlations
 
count() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the number of elements in the RDD.
count() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
The number of edges in the RDD.
count() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
The number of vertices in the RDD.
count() - Method in class org.apache.spark.mllib.evaluation.binary.BinaryConfusionMatrixImpl
 
count() - Method in class org.apache.spark.mllib.recommendation.ALS.BlockStats
 
count() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
 
count() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Sample size.
count() - Method in class org.apache.spark.mllib.tree.impurity.EntropyCalculator
Number of data points accounted for in the sufficient statistics.
count() - Method in class org.apache.spark.mllib.tree.impurity.GiniCalculator
Number of data points accounted for in the sufficient statistics.
count() - Method in class org.apache.spark.mllib.tree.impurity.ImpurityCalculator
Number of data points accounted for in the sufficient statistics.
count() - Method in class org.apache.spark.mllib.tree.impurity.VarianceCalculator
Number of data points accounted for in the sufficient statistics.
count() - Method in class org.apache.spark.rdd.RDD
Return the number of elements in the RDD.
count() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
 
count() - Method in class org.apache.spark.sql.columnar.ColumnStatisticsSchema
 
count() - Method in interface org.apache.spark.sql.columnar.ColumnStats
 
COUNT() - Static method in class org.apache.spark.sql.hive.HiveQl
 
count() - Method in class org.apache.spark.sql.SchemaRDD
:: Experimental :: Return the number of elements in the RDD.
count() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by counting each RDD of this DStream.
count() - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by counting each RDD of this DStream.
count() - Method in class org.apache.spark.util.StatCounter
 
countApprox(long, double) - Method in interface org.apache.spark.api.java.JavaRDDLike
:: Experimental :: Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApprox(long) - Method in interface org.apache.spark.api.java.JavaRDDLike
:: Experimental :: Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApprox(long, double) - Method in class org.apache.spark.rdd.RDD
:: Experimental :: Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApproxDistinct(double) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return approximate number of distinct elements in the RDD.
countApproxDistinct(int, int) - Method in class org.apache.spark.rdd.RDD
:: Experimental :: Return approximate number of distinct elements in the RDD.
countApproxDistinct(double) - Method in class org.apache.spark.rdd.RDD
Return approximate number of distinct elements in the RDD.
countApproxDistinctByKey(double, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(int, int, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
:: Experimental ::
countApproxDistinctByKey(double, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countAsync() - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of count, which returns a future for counting the number of elements in this RDD.
countAsync() - Method in class org.apache.spark.rdd.AsyncRDDActions
Returns a future for counting the number of elements in the RDD.
countByKey() - Method in class org.apache.spark.api.java.JavaPairRDD
Count the number of elements for each key, and return the result to the master as a Map.
countByKey() - Method in class org.apache.spark.rdd.PairRDDFunctions
Count the number of elements for each key, collecting the results to a local Map.
countByKeyApprox(long) - Method in class org.apache.spark.api.java.JavaPairRDD
:: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByKeyApprox(long, double) - Method in class org.apache.spark.api.java.JavaPairRDD
:: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByKeyApprox(long, double) - Method in class org.apache.spark.rdd.PairRDDFunctions
:: Experimental :: Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByValue() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the count of each unique value in this RDD as a map of (value, count) pairs.
countByValue(Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return the count of each unique value in this RDD as a local map of (value, count) pairs.
countByValue() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValue(int) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValue(int, Ordering<T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValueAndWindow(Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueAndWindow(Duration, Duration, int) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueAndWindow(Duration, Duration, int, Ordering<T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueApprox(long, double) - Method in interface org.apache.spark.api.java.JavaRDDLike
(Experimental) Approximate version of countByValue().
countByValueApprox(long) - Method in interface org.apache.spark.api.java.JavaRDDLike
(Experimental) Approximate version of countByValue().
countByValueApprox(long, double, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
:: Experimental :: Approximate version of countByValue().
countByWindow(Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by counting the number of elements in a window over this DStream.
countByWindow(Duration, Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by counting the number of elements in a sliding window over this DStream.
counter() - Method in class org.apache.spark.partial.MeanEvaluator
 
counter() - Method in class org.apache.spark.partial.SumEvaluator
 
CountEvaluator - Class in org.apache.spark.partial
An ApproximateEvaluator for counts.
CountEvaluator(int, double) - Constructor for class org.apache.spark.partial.CountEvaluator
 
cpFile() - Method in class org.apache.spark.rdd.RDDCheckpointData
 
cpRDD() - Method in class org.apache.spark.rdd.RDDCheckpointData
 
cpState() - Method in class org.apache.spark.rdd.RDDCheckpointData
 
CPUS_PER_TASK() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
CR() - Method in class org.apache.spark.ui.ConsoleProgressBar
 
create(boolean, boolean, boolean, int) - Static method in class org.apache.spark.api.java.StorageLevels
Deprecated.
create(boolean, boolean, boolean, boolean, int) - Static method in class org.apache.spark.api.java.StorageLevels
Create a new StorageLevel object.
create(JavaSparkContext, JdbcRDD.ConnectionFactory, String, long, long, int, Function<ResultSet, T>) - Static method in class org.apache.spark.rdd.JdbcRDD
Create an RDD that executes an SQL query on a JDBC connection and reads results.
create(JavaSparkContext, JdbcRDD.ConnectionFactory, String, long, long, int) - Static method in class org.apache.spark.rdd.JdbcRDD
Create an RDD that executes an SQL query on a JDBC connection and reads results.
create(RDD<T>, Function1<Object, Object>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
Create a PartitionPruningRDD.
create(Object...) - Static method in class org.apache.spark.sql.api.java.Row
Creates a Row with the given values.
create(Seq<Object>) - Static method in class org.apache.spark.sql.api.java.Row
Creates a Row with the given values.
create(String, LogicalPlan, Configuration, SQLContext) - Static method in class org.apache.spark.sql.parquet.ParquetRelation
Creates a new ParquetRelation and underlying Parquetfile for the given LogicalPlan.
create() - Method in interface org.apache.spark.streaming.api.java.JavaStreamingContextFactory
 
createActorSystem(String, String, int, SparkConf, SecurityManager) - Static method in class org.apache.spark.util.AkkaUtils
Creates an ActorSystem ready for remoting, with various Spark features.
createArrayType(DataType) - Static method in class org.apache.spark.sql.api.java.DataType
Creates an ArrayType by specifying the data type of elements (elementType).
createArrayType(DataType, boolean) - Static method in class org.apache.spark.sql.api.java.DataType
Creates an ArrayType by specifying the data type of elements (elementType) and whether the array contains null values (containsNull).
createCombiner() - Method in class org.apache.spark.Aggregator
 
createCommand(Protos.Offer, int) - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
createCompiledClass(String, File, String) - Static method in class org.apache.spark.TestUtils
Creates a compiled class with the given name.
createDecimal(BigDecimal) - Static method in class org.apache.spark.sql.hive.HiveShim
 
createDefaultDBIfNeeded(HiveContext) - Static method in class org.apache.spark.sql.hive.HiveShim
 
createDriverEnv(SparkConf, boolean, LiveListenerBus) - Static method in class org.apache.spark.SparkEnv
Create a SparkEnv for the driver.
createDriverResultsArray() - Static method in class org.apache.spark.sql.hive.HiveShim
 
createEmpty(String, Seq<Attribute>, boolean, Configuration, SQLContext) - Static method in class org.apache.spark.sql.parquet.ParquetRelation
Creates an empty ParquetRelation and underlying Parquetfile that only consists of the Metadata for the given schema.
createExecutorEnv(SparkConf, String, String, int, int, boolean, ActorSystem) - Static method in class org.apache.spark.SparkEnv
Create a SparkEnv for an executor.
createExecutorInfo(String) - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
createFilter(Expression) - Static method in class org.apache.spark.sql.parquet.ParquetFilters
 
createFunction() - Method in class org.apache.spark.sql.hive.HiveFunctionWrapper
 
createHistoryUI(SparkConf, SparkListenerBus, SecurityManager, String, String) - Static method in class org.apache.spark.ui.SparkUI
 
createJar(Seq<File>, File) - Static method in class org.apache.spark.TestUtils
Create a jar file that contains this set of files.
createJarWithClasses(Seq<String>, String) - Static method in class org.apache.spark.TestUtils
Create a jar that defines classes with the given names.
createJobID(Date, int) - Static method in class org.apache.spark.SparkHadoopWriter
 
createLiveUI(SparkContext, SparkConf, SparkListenerBus, JobProgressListener, SecurityManager, String) - Static method in class org.apache.spark.ui.SparkUI
 
createMapType(DataType, DataType) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a MapType by specifying the data type of keys (keyType) and values (keyType).
createMapType(DataType, DataType, boolean) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a MapType by specifying the data type of keys (keyType), the data type of values (keyType), and whether values contain any null value (valueContainsNull).
createMesosTask(TaskDescription, String) - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
Turn a Spark TaskDescription into a Mesos task
createMetricsSystem(String, SparkConf, SecurityManager) - Static method in class org.apache.spark.metrics.MetricsSystem
 
createNewSparkContext(SparkConf) - Static method in class org.apache.spark.streaming.StreamingContext
 
createNewSparkContext(String, String, String, Seq<String>, Map<String, String>) - Static method in class org.apache.spark.streaming.StreamingContext
 
createParquetFile(Class<?>, String, boolean, Configuration) - Method in class org.apache.spark.sql.api.java.JavaSQLContext
:: Experimental :: Creates an empty parquet file with the schema of class beanClass, which can be registered as a table.
createParquetFile(String, boolean, Configuration, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLContext
:: Experimental :: Creates an empty parquet file with the schema of class A, which can be registered as a table.
createPathFromString(String, JobConf) - Static method in class org.apache.spark.SparkHadoopWriter
 
createPathFromString(String, JobConf) - Static method in class org.apache.spark.sql.hive.SparkHiveWriterContainer
 
createPlan(String) - Static method in class org.apache.spark.sql.hive.HiveQl
Creates LogicalPlan for a given HiveQL string.
createPlanForView(Table, Option<String>) - Static method in class org.apache.spark.sql.hive.HiveQl
Creates LogicalPlan for a given VIEW
createPollingStream(StreamingContext, String, int, StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(StreamingContext, Seq<InetSocketAddress>, StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(StreamingContext, Seq<InetSocketAddress>, StorageLevel, int, int) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(JavaStreamingContext, String, int) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(JavaStreamingContext, String, int, StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(JavaStreamingContext, InetSocketAddress[], StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPollingStream(JavaStreamingContext, InetSocketAddress[], StorageLevel, int, int) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates an input stream that is to be used with the Spark Sink deployed on a Flume agent.
createPythonWorker(String, Map<String, String>) - Method in class org.apache.spark.SparkEnv
 
createRecordFilter(Seq<Expression>) - Static method in class org.apache.spark.sql.parquet.ParquetFilters
 
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.spark.input.FixedLengthBinaryInputFormat
Create a FixedLengthBinaryRecordReader
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.spark.input.StreamFileInputFormat
 
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.spark.input.StreamInputFormat
 
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.spark.input.WholeTextFileInputFormat
 
createRecordReader(InputSplit, TaskAttemptContext) - Method in class org.apache.spark.sql.parquet.FilteringParquetRowInputFormat
 
createRedirectHandler(String, String, Function1<HttpServletRequest, BoxedUnit>, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a handler that always redirects the user to the given path
createRelation(SQLContext, Map<String, String>) - Method in class org.apache.spark.sql.json.DefaultSource
Returns a new base relation with the given parameters.
createRelation(SQLContext, Map<String, String>) - Method in class org.apache.spark.sql.parquet.DefaultSource
Returns a new base relation with the given parameters.
createRelation(SQLContext, Map<String, String>) - Method in interface org.apache.spark.sql.sources.RelationProvider
Returns a new base relation with the given parameters.
createRoutingTables(EdgeRDD<?>, Partitioner) - Static method in class org.apache.spark.graphx.VertexRDD
 
createSchemaRDD(RDD<A>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLContext
Creates a SchemaRDD from an RDD of case classes.
createServlet(JettyUtils.ServletParams<T>, SecurityManager, Function1<T, Object>) - Static method in class org.apache.spark.ui.JettyUtils
 
createServletHandler(String, JettyUtils.ServletParams<T>, SecurityManager, String, Function1<T, Object>) - Static method in class org.apache.spark.ui.JettyUtils
Create a context handler that responds to a request with the given path prefix
createServletHandler(String, HttpServlet, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a context handler that responds to a request with the given path prefix
createStaticHandler(String, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a handler for serving files from a static directory
createStream(StreamingContext, String, int, StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Create a input stream from a Flume source.
createStream(StreamingContext, String, int, StorageLevel, boolean) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Create a input stream from a Flume source.
createStream(JavaStreamingContext, String, int) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates a input stream from a Flume source.
createStream(JavaStreamingContext, String, int, StorageLevel) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates a input stream from a Flume source.
createStream(JavaStreamingContext, String, int, StorageLevel, boolean) - Static method in class org.apache.spark.streaming.flume.FlumeUtils
Creates a input stream from a Flume source.
createStream(StreamingContext, String, String, Map<String, Object>, StorageLevel) - Static method in class org.apache.spark.streaming.kafka.KafkaUtils
Create an input stream that pulls messages from a Kafka Broker.
createStream(StreamingContext, Map<String, String>, Map<String, Object>, StorageLevel, ClassTag<K>, ClassTag<V>, ClassTag<U>, ClassTag<T>) - Static method in class org.apache.spark.streaming.kafka.KafkaUtils
Create an input stream that pulls messages from a Kafka Broker.
createStream(JavaStreamingContext, String, String, Map<String, Integer>) - Static method in class org.apache.spark.streaming.kafka.KafkaUtils
Create an input stream that pulls messages from a Kafka Broker.
createStream(JavaStreamingContext, String, String, Map<String, Integer>, StorageLevel) - Static method in class org.apache.spark.streaming.kafka.KafkaUtils
Create an input stream that pulls messages from a Kafka Broker.
createStream(JavaStreamingContext, Class<K>, Class<V>, Class<U>, Class<T>, Map<String, String>, Map<String, Integer>, StorageLevel) - Static method in class org.apache.spark.streaming.kafka.KafkaUtils
Create an input stream that pulls messages from a Kafka Broker.
createStream(StreamingContext, String, String, Duration, InitialPositionInStream, StorageLevel) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Create an InputDStream that pulls messages from a Kinesis stream.
createStream(JavaStreamingContext, String, String, Duration, InitialPositionInStream, StorageLevel) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Create a Java-friendly InputDStream that pulls messages from a Kinesis stream.
createStream(StreamingContext, String, String, StorageLevel) - Static method in class org.apache.spark.streaming.mqtt.MQTTUtils
Create an input stream that receives messages pushed by a MQTT publisher.
createStream(JavaStreamingContext, String, String) - Static method in class org.apache.spark.streaming.mqtt.MQTTUtils
Create an input stream that receives messages pushed by a MQTT publisher.
createStream(JavaStreamingContext, String, String, StorageLevel) - Static method in class org.apache.spark.streaming.mqtt.MQTTUtils
Create an input stream that receives messages pushed by a MQTT publisher.
createStream(StreamingContext, Option<Authorization>, Seq<String>, StorageLevel) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter.
createStream(JavaStreamingContext) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter using Twitter4J's default OAuth authentication; this requires the system properties twitter4j.oauth.consumerKey, twitter4j.oauth.consumerSecret, twitter4j.oauth.accessToken and twitter4j.oauth.accessTokenSecret.
createStream(JavaStreamingContext, String[]) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter using Twitter4J's default OAuth authentication; this requires the system properties twitter4j.oauth.consumerKey, twitter4j.oauth.consumerSecret, twitter4j.oauth.accessToken and twitter4j.oauth.accessTokenSecret.
createStream(JavaStreamingContext, String[], StorageLevel) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter using Twitter4J's default OAuth authentication; this requires the system properties twitter4j.oauth.consumerKey, twitter4j.oauth.consumerSecret, twitter4j.oauth.accessToken and twitter4j.oauth.accessTokenSecret.
createStream(JavaStreamingContext, Authorization) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter.
createStream(JavaStreamingContext, Authorization, String[]) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter.
createStream(JavaStreamingContext, Authorization, String[], StorageLevel) - Static method in class org.apache.spark.streaming.twitter.TwitterUtils
Create a input stream that returns tweets received from Twitter.
createStream(StreamingContext, String, Subscribe, Function1<Seq<ByteString>, Iterator<T>>, StorageLevel, SupervisorStrategy, ClassTag<T>) - Static method in class org.apache.spark.streaming.zeromq.ZeroMQUtils
Create an input stream that receives messages pushed by a zeromq publisher.
createStream(JavaStreamingContext, String, Subscribe, Function<byte[][], Iterable<T>>, StorageLevel, SupervisorStrategy) - Static method in class org.apache.spark.streaming.zeromq.ZeroMQUtils
Create an input stream that receives messages pushed by a zeromq publisher.
createStream(JavaStreamingContext, String, Subscribe, Function<byte[][], Iterable<T>>, StorageLevel) - Static method in class org.apache.spark.streaming.zeromq.ZeroMQUtils
Create an input stream that receives messages pushed by a zeromq publisher.
createStream(JavaStreamingContext, String, Subscribe, Function<byte[][], Iterable<T>>) - Static method in class org.apache.spark.streaming.zeromq.ZeroMQUtils
Create an input stream that receives messages pushed by a zeromq publisher.
createStructField(String, DataType, boolean, Metadata) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a StructField by specifying the name (name), data type (dataType) and whether values of this field can be null values (nullable).
createStructField(String, DataType, boolean) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a StructField with empty metadata.
createStructType(List<StructField>) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a StructType with the given list of StructFields (fields).
createStructType(StructField[]) - Static method in class org.apache.spark.sql.api.java.DataType
Creates a StructType with the given StructField array (fields).
createTable(String, boolean, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.hive.HiveContext
Creates a table using the schema of the given class.
createTable(String, String, Seq<Attribute>, boolean, Option<CreateTableDesc>) - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog
Create table with specified database, table name, table description and schema
CreateTableAsSelect - Class in org.apache.spark.sql.hive.execution
:: Experimental :: Create table and insert the query result into it.
CreateTableAsSelect(String, String, LogicalPlan, boolean, Option<CreateTableDesc>) - Constructor for class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
CreateTables() - Method in class org.apache.spark.sql.hive.HiveMetastoreCatalog
 
CreateTableUsing - Class in org.apache.spark.sql.sources
 
CreateTableUsing(String, String, Map<String, String>) - Constructor for class org.apache.spark.sql.sources.CreateTableUsing
 
createTempDir(String) - Static method in class org.apache.spark.util.Utils
Create a temporary directory inside the given parent directory
createTempLocalBlock() - Method in class org.apache.spark.storage.DiskBlockManager
Produces a unique block id and File suitable for storing local intermediate results.
createTempShuffleBlock() - Method in class org.apache.spark.storage.DiskBlockManager
Produces a unique block id and File suitable for storing shuffled intermediate results.
createTime() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
createUsingIndex(Iterator<Product2<Object, VD2>>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexPartitionBaseOps
Similar effect as aggregateUsingIndex((a, b) => a)
createWorkspace(int) - Static method in class org.apache.spark.mllib.optimization.NNLS
 
creationSite() - Method in class org.apache.spark.rdd.RDD
User code that created this RDD (e.g.
creationSite() - Method in class org.apache.spark.streaming.dstream.DStream
 
credentialsProvider() - Method in class org.apache.spark.streaming.kinesis.KinesisReceiver
 
CrossValidator - Class in org.apache.spark.ml.tuning
:: AlphaComponent :: K-fold cross validation.
CrossValidator() - Constructor for class org.apache.spark.ml.tuning.CrossValidator
 
CrossValidatorModel - Class in org.apache.spark.ml.tuning
:: AlphaComponent :: Model from k-fold cross validation.
CrossValidatorModel(CrossValidator, ParamMap, Model<?>) - Constructor for class org.apache.spark.ml.tuning.CrossValidatorModel
 
CrossValidatorParams - Interface in org.apache.spark.ml.tuning
CSV_DEFAULT_DIR() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CSV_DEFAULT_PERIOD() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CSV_DEFAULT_UNIT() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CSV_KEY_DIR() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CSV_KEY_PERIOD() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CSV_KEY_UNIT() - Method in class org.apache.spark.metrics.sink.CsvSink
 
CsvSink - Class in org.apache.spark.metrics.sink
 
CsvSink(Properties, MetricRegistry, SecurityManager) - Constructor for class org.apache.spark.metrics.sink.CsvSink
 
currentAttemptId() - Method in interface org.apache.spark.SparkStageInfo
 
currentAttemptId() - Method in class org.apache.spark.SparkStageInfoImpl
 
currentInterval(Duration) - Static method in class org.apache.spark.streaming.Interval
 
currentLocalityIndex() - Method in class org.apache.spark.scheduler.TaskSetManager
 
currentResult() - Method in interface org.apache.spark.partial.ApproximateEvaluator
 
currentResult() - Method in class org.apache.spark.partial.CountEvaluator
 
currentResult() - Method in class org.apache.spark.partial.GroupedCountEvaluator
 
currentResult() - Method in class org.apache.spark.partial.GroupedMeanEvaluator
 
currentResult() - Method in class org.apache.spark.partial.GroupedSumEvaluator
 
currentResult() - Method in class org.apache.spark.partial.MeanEvaluator
 
currentResult() - Method in class org.apache.spark.partial.SumEvaluator
 
currentTime() - Method in interface org.apache.spark.streaming.util.Clock
 
currentTime() - Method in class org.apache.spark.streaming.util.ManualClock
 
currentTime() - Method in class org.apache.spark.streaming.util.SystemClock
 
currentUnrollMemory() - Method in class org.apache.spark.storage.MemoryStore
Return the amount of memory currently occupied for unrolling blocks across all threads.
currentUnrollMemoryForThisThread() - Method in class org.apache.spark.storage.MemoryStore
Return the amount of memory currently occupied for unrolling blocks by this thread.
currPrefLocs(Partition) - Method in class org.apache.spark.rdd.PartitionCoalescer
 

D

DAGScheduler - Class in org.apache.spark.scheduler
The high-level scheduling layer that implements stage-oriented scheduling.
DAGScheduler(SparkContext, TaskScheduler, LiveListenerBus, MapOutputTrackerMaster, BlockManagerMaster, SparkEnv, Clock) - Constructor for class org.apache.spark.scheduler.DAGScheduler
 
DAGScheduler(SparkContext, TaskScheduler) - Constructor for class org.apache.spark.scheduler.DAGScheduler
 
DAGScheduler(SparkContext) - Constructor for class org.apache.spark.scheduler.DAGScheduler
 
dagScheduler() - Method in class org.apache.spark.scheduler.DAGSchedulerSource
 
dagScheduler() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
dagScheduler() - Method in class org.apache.spark.SparkContext
 
DAGSchedulerActorSupervisor - Class in org.apache.spark.scheduler
 
DAGSchedulerActorSupervisor(DAGScheduler) - Constructor for class org.apache.spark.scheduler.DAGSchedulerActorSupervisor
 
DAGSchedulerEvent - Interface in org.apache.spark.scheduler
Types of events that can be handled by the DAGScheduler.
DAGSchedulerEventProcessActor - Class in org.apache.spark.scheduler
 
DAGSchedulerEventProcessActor(DAGScheduler) - Constructor for class org.apache.spark.scheduler.DAGSchedulerEventProcessActor
 
DAGSchedulerSource - Class in org.apache.spark.scheduler
 
DAGSchedulerSource(DAGScheduler) - Constructor for class org.apache.spark.scheduler.DAGSchedulerSource
 
data() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask
 
data() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
data() - Method in class org.apache.spark.storage.BlockResult
 
data() - Method in class org.apache.spark.storage.PutResult
 
data() - Method in class org.apache.spark.util.Distribution
 
data() - Method in class org.apache.spark.util.random.GapSamplingIterator
 
data() - Method in class org.apache.spark.util.random.GapSamplingReplacementIterator
 
database() - Method in class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
databaseName() - Method in class org.apache.spark.sql.hive.MetastoreRelation
 
dataDeserialize(BlockId, ByteBuffer, Serializer) - Method in class org.apache.spark.storage.BlockManager
Deserializes a ByteBuffer into an iterator of values and disposes of it when the end of the iterator is reached.
dataIncludesKey() - Method in class org.apache.spark.sql.parquet.ParquetRelation2
 
dataSchema() - Method in class org.apache.spark.sql.parquet.ParquetRelation2
 
dataSerialize(BlockId, Iterator<Object>, Serializer) - Method in class org.apache.spark.storage.BlockManager
Serializes into a byte buffer.
dataSerializeStream(BlockId, OutputStream, Iterator<Object>, Serializer) - Method in class org.apache.spark.storage.BlockManager
Serializes into a stream.
DataSinks() - Method in interface org.apache.spark.sql.hive.HiveStrategies
 
DataSourceStrategy - Class in org.apache.spark.sql.sources
A Strategy for planning scans over data sources defined using the sources API.
DataSourceStrategy() - Constructor for class org.apache.spark.sql.sources.DataSourceStrategy
 
DataType - Class in org.apache.spark.sql.api.java
The base type of all Spark SQL data types.
DataType() - Constructor for class org.apache.spark.sql.api.java.DataType
 
dataType() - Method in class org.apache.spark.sql.columnar.NativeColumnType
 
dataType() - Method in class org.apache.spark.sql.execution.PythonUDF
 
dataType() - Method in class org.apache.spark.sql.hive.HiveGenericUdaf
 
dataType() - Method in class org.apache.spark.sql.hive.HiveGenericUdf
 
dataType() - Method in class org.apache.spark.sql.hive.HiveSimpleUdf
 
dataType() - Method in class org.apache.spark.sql.hive.HiveUdaf
 
DataTypeConversions - Class in org.apache.spark.sql.types.util
 
DataTypeConversions() - Constructor for class org.apache.spark.sql.types.util.DataTypeConversions
 
DataValidators - Class in org.apache.spark.mllib.util
:: DeveloperApi :: A collection of methods used to validate data before applying ML algorithms.
DataValidators() - Constructor for class org.apache.spark.mllib.util.DataValidators
 
DATE - Class in org.apache.spark.sql.columnar
 
DATE() - Constructor for class org.apache.spark.sql.columnar.DATE
 
DateColumnAccessor - Class in org.apache.spark.sql.columnar
 
DateColumnAccessor(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.DateColumnAccessor
 
DateColumnBuilder - Class in org.apache.spark.sql.columnar
 
DateColumnBuilder() - Constructor for class org.apache.spark.sql.columnar.DateColumnBuilder
 
DateColumnStats - Class in org.apache.spark.sql.columnar
 
DateColumnStats() - Constructor for class org.apache.spark.sql.columnar.DateColumnStats
 
DateType - Static variable in class org.apache.spark.sql.api.java.DataType
Gets the DateType object.
DateType - Class in org.apache.spark.sql.api.java
The data type representing java.sql.Date values.
datum() - Method in class org.apache.spark.mllib.tree.impl.BaggedPoint
 
DDLParser - Class in org.apache.spark.sql.sources
A parser for foreign DDL commands.
DDLParser() - Constructor for class org.apache.spark.sql.sources.DDLParser
 
dead(String) - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
decayFactor() - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
decimalMetadata() - Method in class org.apache.spark.sql.parquet.ParquetTypeInfo
 
decimalMetastoreString(DecimalType) - Static method in class org.apache.spark.sql.hive.HiveShim
 
DecimalType - Class in org.apache.spark.sql.api.java
The data type representing java.math.BigDecimal values.
DecimalType(int, int) - Constructor for class org.apache.spark.sql.api.java.DecimalType
 
DecimalType() - Constructor for class org.apache.spark.sql.api.java.DecimalType
 
decimalTypeInfo(DecimalType) - Static method in class org.apache.spark.sql.hive.HiveShim
 
decimalTypeInfoToCatalyst(PrimitiveObjectInspector) - Static method in class org.apache.spark.sql.hive.HiveShim
 
DecisionTree - Class in org.apache.spark.mllib.tree
:: Experimental :: A class which implements a decision tree learning algorithm for classification and regression.
DecisionTree(Strategy) - Constructor for class org.apache.spark.mllib.tree.DecisionTree
 
DecisionTreeMetadata - Class in org.apache.spark.mllib.tree.impl
Learning and dataset metadata for DecisionTree.
DecisionTreeMetadata(int, long, int, int, Map<Object, Object>, Set<Object>, int[], Impurity, Enumeration.Value, int, int, double, int, int) - Constructor for class org.apache.spark.mllib.tree.impl.DecisionTreeMetadata
 
DecisionTreeModel - Class in org.apache.spark.mllib.tree.model
:: Experimental :: Decision tree model for classification or regression.
DecisionTreeModel(Node, Enumeration.Value) - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.BooleanBitSet
 
decoder() - Method in interface org.apache.spark.sql.columnar.compression.CompressibleColumnAccessor
 
decoder(ByteBuffer, NativeColumnType<T>) - Method in interface org.apache.spark.sql.columnar.compression.CompressionScheme
 
Decoder<T extends org.apache.spark.sql.catalyst.types.NativeType> - Interface in org.apache.spark.sql.columnar.compression
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.DictionaryEncoding
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.IntDelta
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.LongDelta
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.PassThrough
 
decoder(ByteBuffer, NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.RunLengthEncoding
 
decreaseRunningTasks(int) - Method in class org.apache.spark.scheduler.Pool
 
deepCopy() - Method in class org.apache.spark.mllib.tree.model.Node
Returns a deep copy of the subtree rooted at this node.
DEFAULT_BUFFER_SIZE() - Static method in class org.apache.spark.util.logging.RollingFileAppender
 
DEFAULT_CLEANER_TTL() - Static method in class org.apache.spark.streaming.StreamingContext
 
DEFAULT_LOG_DIR() - Static method in class org.apache.spark.scheduler.EventLoggingListener
 
DEFAULT_MINIMUM_SHARE() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
DEFAULT_POOL_NAME() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
DEFAULT_POOL_NAME() - Static method in class org.apache.spark.ui.jobs.JobProgressListener
 
DEFAULT_PORT() - Static method in class org.apache.spark.ui.SparkUI
 
DEFAULT_PREFIX() - Method in class org.apache.spark.metrics.MetricsConfig
 
DEFAULT_RETAINED_JOBS() - Static method in class org.apache.spark.ui.jobs.JobProgressListener
 
DEFAULT_RETAINED_STAGES() - Static method in class org.apache.spark.ui.jobs.JobProgressListener
 
DEFAULT_SCHEDULER_FILE() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
DEFAULT_SCHEDULING_MODE() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
DEFAULT_WEIGHT() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
defaultCorrName() - Static method in class org.apache.spark.mllib.stat.correlation.CorrelationNames
 
defaultFilter(Path) - Static method in class org.apache.spark.streaming.dstream.FileInputDStream
 
defaultMinPartitions() - Method in class org.apache.spark.api.java.JavaSparkContext
Default min number of partitions for Hadoop RDDs when not given by user
defaultMinPartitions() - Method in class org.apache.spark.SparkContext
Default min number of partitions for Hadoop RDDs when not given by user
defaultMinSplits() - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
As of Spark 1.0.0, defaultMinSplits is deprecated, use JavaSparkContext.defaultMinPartitions() instead
defaultMinSplits() - Method in class org.apache.spark.SparkContext
Default min number of partitions for Hadoop RDDs when not given by user
defaultParallelism() - Method in class org.apache.spark.api.java.JavaSparkContext
Default level of parallelism to use when not given by user (e.g.
defaultParallelism() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
defaultParallelism() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
defaultParallelism() - Method in class org.apache.spark.scheduler.local.LocalBackend
 
defaultParallelism() - Method in interface org.apache.spark.scheduler.SchedulerBackend
 
defaultParallelism() - Method in interface org.apache.spark.scheduler.TaskScheduler
 
defaultParallelism() - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
defaultParallelism() - Method in class org.apache.spark.SparkContext
Default level of parallelism to use when not given by user (e.g.
defaultParams(String) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
Returns default configuration for the boosting algorithm
defaultPartitioner(RDD<?>, Seq<RDD<?>>) - Static method in class org.apache.spark.Partitioner
Choose a partitioner to use for a cogroup-like operation between a number of RDDs.
defaultPartitioner(int) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
 
defaultProbabilities() - Method in class org.apache.spark.util.Distribution
 
defaultSize() - Method in class org.apache.spark.sql.columnar.ColumnType
 
defaultSizeInBytes() - Method in interface org.apache.spark.sql.SQLConf
The default size in bytes to assign to a logical operator's estimation statistics.
DefaultSource - Class in org.apache.spark.sql.json
 
DefaultSource() - Constructor for class org.apache.spark.sql.json.DefaultSource
 
DefaultSource - Class in org.apache.spark.sql.parquet
Allows creation of parquet based tables using the syntax CREATE TEMPORARY TABLE ...
DefaultSource() - Constructor for class org.apache.spark.sql.parquet.DefaultSource
 
defaultStrategy(String) - Static method in class org.apache.spark.mllib.tree.configuration.Strategy
Construct a default set of parameters for DecisionTree
defaultStrategy() - Static method in class org.apache.spark.streaming.receiver.ActorSupervisorStrategy
 
defaultValue() - Method in class org.apache.spark.ml.param.Param
 
DeferredObjectAdapter - Class in org.apache.spark.sql.hive
 
DeferredObjectAdapter(ObjectInspector) - Constructor for class org.apache.spark.sql.hive.DeferredObjectAdapter
 
degrees() - Method in class org.apache.spark.graphx.GraphOps
The degree of each vertex in the graph.
degreesOfFreedom() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
degreesOfFreedom() - Method in interface org.apache.spark.mllib.stat.test.TestResult
Returns the degree(s) of freedom of the hypothesis test.
delaySeconds() - Method in class org.apache.spark.streaming.Checkpoint
 
delegate() - Method in class org.apache.spark.InterruptibleIterator
 
deleteAllCheckpoints() - Method in class org.apache.spark.mllib.tree.impl.NodeIdCache
Call this after training is finished to delete any remaining checkpoints.
deleteOldFiles() - Method in class org.apache.spark.util.logging.RollingFileAppender
Retain only last few files
deleteRecursively(File) - Static method in class org.apache.spark.util.Utils
Delete a file or directory and its contents recursively.
deleteRecursively(TachyonFile, TachyonFS) - Static method in class org.apache.spark.util.Utils
Delete a file or directory and its contents recursively.
dense(int, int, double[]) - Static method in class org.apache.spark.mllib.linalg.Matrices
Creates a column-major dense matrix.
dense(double, double...) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from its values.
dense(double, Seq<Object>) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from its values.
dense(double[]) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from a double array.
DenseMatrix - Class in org.apache.spark.mllib.linalg
Column-major dense matrix.
DenseMatrix(int, int, double[]) - Constructor for class org.apache.spark.mllib.linalg.DenseMatrix
 
DenseVector - Class in org.apache.spark.mllib.linalg
A dense vector represented by a value array.
DenseVector(double[]) - Constructor for class org.apache.spark.mllib.linalg.DenseVector
 
dependencies() - Method in class org.apache.spark.rdd.RDD
Get the list of dependencies of this RDD, taking into account whether the RDD is checkpointed or not.
dependencies() - Method in class org.apache.spark.streaming.dstream.DStream
List of parent DStreams on which this DStream depends on
dependencies() - Method in class org.apache.spark.streaming.dstream.FilteredDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.FlatMappedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.FlatMapValuedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.ForEachDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.GlommedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.InputDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.MapPartitionedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.MappedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.MapValuedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.ReducedWindowedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.ShuffledDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.StateDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.TransformedDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.UnionDStream
 
dependencies() - Method in class org.apache.spark.streaming.dstream.WindowedDStream
 
Dependency<T> - Class in org.apache.spark
:: DeveloperApi :: Base class for dependencies.
Dependency() - Constructor for class org.apache.spark.Dependency
 
deps() - Method in class org.apache.spark.rdd.CoGroupPartition
 
depth() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Get depth of tree.
DeregisterReceiver - Class in org.apache.spark.streaming.scheduler
 
DeregisterReceiver(int, String, String) - Constructor for class org.apache.spark.streaming.scheduler.DeregisterReceiver
 
desc() - Method in class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
DescribeCommand - Class in org.apache.spark.sql.execution
:: DeveloperApi ::
DescribeCommand(SparkPlan, Seq<Attribute>, SQLContext) - Constructor for class org.apache.spark.sql.execution.DescribeCommand
 
describedTable() - Method in class org.apache.spark.sql.hive.test.TestHiveContext
 
DescribeHiveTableCommand - Class in org.apache.spark.sql.hive.execution
Implementation for "describe [extended] table".
DescribeHiveTableCommand(MetastoreRelation, Seq<Attribute>, boolean, HiveContext) - Constructor for class org.apache.spark.sql.hive.execution.DescribeHiveTableCommand
 
description() - Method in class org.apache.spark.ExceptionFailure
 
description() - Method in class org.apache.spark.storage.StorageLevel
 
description() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
DeserializationStream - Class in org.apache.spark.serializer
:: DeveloperApi :: A stream for reading serialized objects.
DeserializationStream() - Constructor for class org.apache.spark.serializer.DeserializationStream
 
deserialize(Object) - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
deserialize(ByteBuffer, ClassTag<T>) - Method in class org.apache.spark.serializer.JavaSerializerInstance
 
deserialize(ByteBuffer, ClassLoader, ClassTag<T>) - Method in class org.apache.spark.serializer.JavaSerializerInstance
 
deserialize(ByteBuffer, ClassTag<T>) - Method in class org.apache.spark.serializer.KryoSerializerInstance
 
deserialize(ByteBuffer, ClassLoader, ClassTag<T>) - Method in class org.apache.spark.serializer.KryoSerializerInstance
 
deserialize(ByteBuffer, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserialize(ByteBuffer, ClassLoader, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserialize(Object) - Method in class org.apache.spark.sql.api.java.JavaToScalaUDTWrapper
Convert a SQL datum to the user type
deserialize(Object) - Method in class org.apache.spark.sql.api.java.ScalaToJavaUDTWrapper
Convert a SQL datum to the user type
deserialize(Object) - Method in class org.apache.spark.sql.api.java.UserDefinedType
Convert a SQL datum to the user type
deserialize(byte[], ClassTag<T>) - Static method in class org.apache.spark.sql.execution.SparkSqlSerializer
 
deserialize(Writable) - Method in class org.apache.spark.sql.hive.parquet.FakeParquetSerDe
 
deserialize(Object) - Method in class org.apache.spark.sql.test.ExamplePointUDT
 
deserialize(byte[]) - Static method in class org.apache.spark.util.Utils
Deserialize an object using Java serialization
deserialize(byte[], ClassLoader) - Static method in class org.apache.spark.util.Utils
Deserialize an object using Java serialization and the given ClassLoader
deserialized() - Method in class org.apache.spark.storage.MemoryEntry
 
deserialized() - Method in class org.apache.spark.storage.StorageLevel
 
deserializeFilterExpressions(Configuration) - Static method in class org.apache.spark.sql.parquet.ParquetFilters
Note: Inside the Hadoop API we only have access to Configuration, not to SparkContext, so we cannot use broadcasts to convey the actual filter predicate.
deserializeLongValue(byte[]) - Static method in class org.apache.spark.util.Utils
Deserialize a Long value (used for PythonPartitioner)
deserializeMapStatuses(byte[]) - Static method in class org.apache.spark.MapOutputTracker
 
deserializePlan(InputStream, Class<?>) - Method in class org.apache.spark.sql.hive.HiveFunctionWrapper
 
deserializeStream(InputStream) - Method in class org.apache.spark.serializer.JavaSerializerInstance
 
deserializeStream(InputStream, ClassLoader) - Method in class org.apache.spark.serializer.JavaSerializerInstance
 
deserializeStream(InputStream) - Method in class org.apache.spark.serializer.KryoSerializerInstance
 
deserializeStream(InputStream) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserializeViaNestedStream(InputStream, SerializerInstance, Function1<DeserializationStream, BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Deserialize via nested stream using specific serializer
deserializeWithDependencies(ByteBuffer) - Static method in class org.apache.spark.scheduler.Task
Deserialize the list of dependencies in a task serialized with serializeWithDependencies, and return the task itself as a serialized ByteBuffer.
destinationToken() - Static method in class org.apache.spark.sql.hive.HiveQl
 
destroy() - Method in class org.apache.spark.broadcast.Broadcast
Destroy all data and metadata related to this broadcast variable.
destroy(boolean) - Method in class org.apache.spark.broadcast.Broadcast
Destroy all data and metadata related to this broadcast variable.
destroyPythonWorker(String, Map<String, String>, Socket) - Method in class org.apache.spark.SparkEnv
 
destTableId() - Method in class org.apache.spark.sql.hive.ShimFileSinkDesc
 
details() - Method in class org.apache.spark.scheduler.Stage
 
details() - Method in class org.apache.spark.scheduler.StageInfo
 
determineBounds(ArrayBuffer<Tuple2<K, Object>>, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.RangePartitioner
Determines the bounds for range partitioning from candidates with weights indicating how many items each represents.
DeveloperApi - Annotation Type in org.apache.spark.annotation
A lower-level, unstable API intended for developers.
diag(Vector) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a diagonal matrix in DenseMatrix format from the supplied values.
dialect() - Method in class org.apache.spark.sql.hive.HiveContext
 
dialect() - Method in interface org.apache.spark.sql.SQLConf
The SQL dialect that is used when parsing queries.
DictionaryEncoding - Class in org.apache.spark.sql.columnar.compression
 
DictionaryEncoding() - Constructor for class org.apache.spark.sql.columnar.compression.DictionaryEncoding
 
DictionaryEncoding.Decoder<T extends org.apache.spark.sql.catalyst.types.NativeType> - Class in org.apache.spark.sql.columnar.compression
 
DictionaryEncoding.Decoder(ByteBuffer, NativeColumnType<T>) - Constructor for class org.apache.spark.sql.columnar.compression.DictionaryEncoding.Decoder
 
DictionaryEncoding.Encoder<T extends org.apache.spark.sql.catalyst.types.NativeType> - Class in org.apache.spark.sql.columnar.compression
 
DictionaryEncoding.Encoder(NativeColumnType<T>) - Constructor for class org.apache.spark.sql.columnar.compression.DictionaryEncoding.Encoder
 
diff(Self) - Method in class org.apache.spark.graphx.impl.VertexPartitionBaseOps
Hides vertices that are the same between this and other.
diff(VertexRDD<VD>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
diff(VertexRDD<VD>) - Method in class org.apache.spark.graphx.VertexRDD
Hides vertices that are the same between this and other; for vertices that are different, keeps the values from other.
dir() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
dir() - Method in class org.apache.spark.sql.hive.ShimFileSinkDesc
 
DirectTaskResult<T> - Class in org.apache.spark.scheduler
A TaskResult that contains the task's return value and accumulator updates.
DirectTaskResult(ByteBuffer, Map<Object, Object>, TaskMetrics) - Constructor for class org.apache.spark.scheduler.DirectTaskResult
 
DirectTaskResult() - Constructor for class org.apache.spark.scheduler.DirectTaskResult
 
disconnected(SchedulerDriver) - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
disconnected(SchedulerDriver) - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
disconnected() - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
DISK_ONLY - Static variable in class org.apache.spark.api.java.StorageLevels
 
DISK_ONLY() - Static method in class org.apache.spark.storage.StorageLevel
 
DISK_ONLY_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
DISK_ONLY_2() - Static method in class org.apache.spark.storage.StorageLevel
 
diskBlockManager() - Method in class org.apache.spark.storage.BlockManager
 
DiskBlockManager - Class in org.apache.spark.storage
Creates and maintains the logical mapping between logical blocks and physical on-disk locations.
DiskBlockManager(BlockManager, SparkConf) - Constructor for class org.apache.spark.storage.DiskBlockManager
 
DiskBlockObjectWriter - Class in org.apache.spark.storage
BlockObjectWriter which writes directly to a file on disk.
DiskBlockObjectWriter(BlockId, File, Serializer, int, Function1<OutputStream, OutputStream>, boolean, ShuffleWriteMetrics) - Constructor for class org.apache.spark.storage.DiskBlockObjectWriter
 
diskBytesSpilled() - Method in class org.apache.spark.ui.jobs.UIData.ExecutorSummary
 
diskBytesSpilled() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
diskSize() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
diskSize() - Method in class org.apache.spark.storage.BlockStatus
 
diskSize() - Method in class org.apache.spark.storage.RDDInfo
 
diskStore() - Method in class org.apache.spark.storage.BlockManager
 
DiskStore - Class in org.apache.spark.storage
Stores BlockManager blocks on disk.
DiskStore(BlockManager, DiskBlockManager) - Constructor for class org.apache.spark.storage.DiskStore
 
diskUsed() - Method in class org.apache.spark.storage.StorageStatus
Return the disk space used by this block manager.
diskUsed() - Method in class org.apache.spark.ui.exec.ExecutorSummaryInfo
 
diskUsedByRdd(int) - Method in class org.apache.spark.storage.StorageStatus
Return the disk space used by the given RDD in this block manager in O(1) time.
dispose(ByteBuffer) - Static method in class org.apache.spark.storage.BlockManager
Attempt to clean up a ByteBuffer if it is memory-mapped.
dist(Vector) - Method in class org.apache.spark.util.Vector
 
distinct() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.rdd.RDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.sql.api.java.JavaSchemaRDD
Return a new RDD containing the distinct elements in this RDD.
Distinct - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Computes the set of distinct input rows using a HashSet.
Distinct(boolean, SparkPlan) - Constructor for class org.apache.spark.sql.execution.Distinct
 
distinct() - Method in class org.apache.spark.sql.SchemaRDD
 
distinct(int, Ordering<Row>) - Method in class org.apache.spark.sql.SchemaRDD
 
DistributedMatrix - Interface in org.apache.spark.mllib.linalg.distributed
Represents a distributively stored matrix backed by one or more RDDs.
Distribution - Class in org.apache.spark.util
Util for getting some stats from a small sample of numeric values, with some handy summary functions.
Distribution(double[], int, int) - Constructor for class org.apache.spark.util.Distribution
 
Distribution(Traversable<Object>) - Constructor for class org.apache.spark.util.Distribution
 
DIV() - Static method in class org.apache.spark.sql.hive.HiveQl
 
div(Duration) - Method in class org.apache.spark.streaming.Duration
 
divide(double) - Method in class org.apache.spark.util.Vector
 
doc() - Method in class org.apache.spark.ml.param.Param
 
doCancelAllJobs() - Method in class org.apache.spark.scheduler.DAGScheduler
 
doCheckpoint() - Method in class org.apache.spark.rdd.RDD
Performs the checkpointing of this RDD by saving this.
doCheckpoint() - Method in class org.apache.spark.rdd.RDDCheckpointData
 
DoCheckpoint - Class in org.apache.spark.streaming.scheduler
 
DoCheckpoint(Time) - Constructor for class org.apache.spark.streaming.scheduler.DoCheckpoint
 
doCleanupBroadcast(long, boolean) - Method in class org.apache.spark.ContextCleaner
Perform broadcast cleanup.
doCleanupRDD(int, boolean) - Method in class org.apache.spark.ContextCleaner
Perform RDD cleanup.
doCleanupShuffle(int, boolean) - Method in class org.apache.spark.ContextCleaner
Perform shuffle cleanup, asynchronously.
doesDirectoryContainAnyNewFiles(File, long) - Static method in class org.apache.spark.util.Utils
Determines if a directory contains any files newer than cutoff seconds.
doKillExecutors(Seq<String>) - Method in class org.apache.spark.scheduler.cluster.YarnSchedulerBackend
Request that the ApplicationMaster kill the specified executors.
doRequestTotalExecutors(int) - Method in class org.apache.spark.scheduler.cluster.YarnSchedulerBackend
Request executors from the ApplicationMaster by specifying the total number desired.
dot(Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
dot(x, y)
dot(Vector) - Method in class org.apache.spark.util.Vector
 
DOUBLE - Class in org.apache.spark.sql.columnar
 
DOUBLE() - Constructor for class org.apache.spark.sql.columnar.DOUBLE
 
doubleAccumulator(double) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator double variable, which tasks can "add" values to using the add method.
doubleAccumulator(double, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Create an Accumulator double variable, which tasks can "add" values to using the add method.
DoubleColumnAccessor - Class in org.apache.spark.sql.columnar
 
DoubleColumnAccessor(ByteBuffer) - Constructor for class org.apache.spark.sql.columnar.DoubleColumnAccessor
 
DoubleColumnBuilder - Class in org.apache.spark.sql.columnar
 
DoubleColumnBuilder() - Constructor for class org.apache.spark.sql.columnar.DoubleColumnBuilder
 
DoubleColumnStats - Class in org.apache.spark.sql.columnar
 
DoubleColumnStats() - Constructor for class org.apache.spark.sql.columnar.DoubleColumnStats
 
DoubleFlatMapFunction<T> - Interface in org.apache.spark.api.java.function
A function that returns zero or more records of type Double from each input record.
DoubleFunction<T> - Interface in org.apache.spark.api.java.function
A function that returns Doubles, and can be used to construct DoubleRDDs.
DoubleParam - Class in org.apache.spark.ml.param
Specialized version of Param[Double] for Java.
DoubleParam(Params, String, String, Option<Object>) - Constructor for class org.apache.spark.ml.param.DoubleParam
 
DoubleRDDFunctions - Class in org.apache.spark.rdd
Extra functions available on RDDs of Doubles through an implicit conversion.
DoubleRDDFunctions(RDD<Object>) - Constructor for class org.apache.spark.rdd.DoubleRDDFunctions
 
doubleRDDToDoubleRDDFunctions(RDD<Object>) - Static method in class org.apache.spark.SparkContext
 
doubleToDoubleWritable(double) - Static method in class org.apache.spark.SparkContext
 
doubleToMultiplier(double) - Static method in class org.apache.spark.util.Vector
 
DoubleType - Static variable in class org.apache.spark.sql.api.java.DataType
Gets the DoubleType object.
DoubleType - Class in org.apache.spark.sql.api.java
The data type representing double and Double values.
doubleWritableConverter() - Static method in class org.apache.spark.SparkContext
 
driver() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
driver() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
DRIVER_AKKA_ACTOR_NAME() - Method in class org.apache.spark.storage.BlockManagerMaster
 
DRIVER_IDENTIFIER() - Static method in class org.apache.spark.SparkContext
 
driverActor() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend
 
driverActor() - Method in class org.apache.spark.storage.BlockManagerMaster
 
driverActorSystemName() - Static method in class org.apache.spark.SparkEnv
 
driverSideSetup() - Method in class org.apache.spark.sql.hive.SparkHiveWriterContainer
 
dropFromMemory(BlockId, Either<Object[], ByteBuffer>) - Method in class org.apache.spark.storage.BlockManager
Drop a block from memory, possibly putting it on disk if applicable.
droppedBlocks() - Method in class org.apache.spark.storage.PutResult
 
droppedBlocks() - Method in class org.apache.spark.storage.ResultWithDroppedBlocks
 
DropTable - Class in org.apache.spark.sql.hive
 
DropTable(String, boolean) - Constructor for class org.apache.spark.sql.hive.DropTable
 
DropTable - Class in org.apache.spark.sql.hive.execution
:: DeveloperApi :: Drops a table from the metastore and removes it if it is cached.
DropTable(String, boolean) - Constructor for class org.apache.spark.sql.hive.execution.DropTable
 
dropTempTable(String) - Method in class org.apache.spark.sql.SQLContext
Drops the temporary table with the given table name in the catalog.
Dst - Static variable in class org.apache.spark.graphx.TripletFields
Expose the destination and edge fields but not the source field.
dstAttr() - Method in class org.apache.spark.graphx.EdgeContext
The vertex attribute of the edge's destination vertex.
dstAttr() - Method in class org.apache.spark.graphx.EdgeTriplet
The destination vertex attribute
dstAttr() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
dstId() - Method in class org.apache.spark.graphx.Edge
 
dstId() - Method in class org.apache.spark.graphx.EdgeContext
The vertex id of the edge's destination vertex.
dstId() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
dstId() - Method in class org.apache.spark.graphx.impl.EdgeWithLocalIds
 
dstream() - Method in class org.apache.spark.streaming.api.java.JavaDStream
 
dstream() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
dstream() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
DStream<T> - Class in org.apache.spark.streaming.dstream
A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data (see org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs).
DStream(StreamingContext, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.DStream
 
DStreamCheckpointData<T> - Class in org.apache.spark.streaming.dstream
 
DStreamCheckpointData(DStream<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.DStreamCheckpointData
 
DStreamGraph - Class in org.apache.spark.streaming
 
DStreamGraph() - Constructor for class org.apache.spark.streaming.DStreamGraph
 
DTStatsAggregator - Class in org.apache.spark.mllib.tree.impl
DecisionTree statistics aggregator for a node.
DTStatsAggregator(DecisionTreeMetadata, Option<int[]>) - Constructor for class org.apache.spark.mllib.tree.impl.DTStatsAggregator
 
DummyCategoricalSplit - Class in org.apache.spark.mllib.tree.model
Split with no acceptable feature values for categorical features.
DummyCategoricalSplit(int, Enumeration.Value) - Constructor for class org.apache.spark.mllib.tree.model.DummyCategoricalSplit
 
DummyHighSplit - Class in org.apache.spark.mllib.tree.model
Split with maximum threshold for continuous features.
DummyHighSplit(int, Enumeration.Value) - Constructor for class org.apache.spark.mllib.tree.model.DummyHighSplit
 
DummyLowSplit - Class in org.apache.spark.mllib.tree.model
Split with minimum threshold for continuous features.
DummyLowSplit(int, Enumeration.Value) - Constructor for class org.apache.spark.mllib.tree.model.DummyLowSplit
 
dumpTree(Node, StringBuilder, int) - Static method in class org.apache.spark.sql.hive.HiveQl
 
duration() - Method in class org.apache.spark.scheduler.TaskInfo
 
Duration - Class in org.apache.spark.streaming
 
Duration(long) - Constructor for class org.apache.spark.streaming.Duration
 
duration() - Method in class org.apache.spark.streaming.Interval
 
Durations - Class in org.apache.spark.streaming
 
Durations() - Constructor for class org.apache.spark.streaming.Durations
 

E

e() - Method in class org.apache.spark.storage.ShuffleBlockFetcherIterator.FailureFetchResult
 
e() - Method in class org.apache.spark.streaming.scheduler.ErrorReported
 
Edge<ED> - Class in org.apache.spark.graphx
A single directed edge consisting of a source id, target id, and the data associated with the edge.
Edge(long, long, ED) - Constructor for class org.apache.spark.graphx.Edge
 
EdgeActiveness - Enum in org.apache.spark.graphx.impl
Criteria for filtering edges based on activeness.
edgeArraySortDataFormat() - Static method in class org.apache.spark.graphx.Edge
 
edgeArraySortDataFormat() - Static method in class org.apache.spark.graphx.impl.EdgeWithLocalIds
 
EdgeContext<VD,ED,A> - Class in org.apache.spark.graphx
Represents an edge along with its neighboring vertices and allows sending messages along the edge.
EdgeContext() - Constructor for class org.apache.spark.graphx.EdgeContext
 
EdgeDirection - Class in org.apache.spark.graphx
The direction of a directed edge relative to a vertex.
edgeListFile(SparkContext, String, boolean, int, StorageLevel, StorageLevel) - Static method in class org.apache.spark.graphx.GraphLoader
Loads a graph from an edge list formatted file where each line contains two integers: a source id and a target id.
EdgeOnly - Static variable in class org.apache.spark.graphx.TripletFields
Expose only the edge field and not the source or destination field.
EdgePartition<ED,VD> - Class in org.apache.spark.graphx.impl
A collection of edges, along with referenced vertex attributes and an optional active vertex set for filtering computation on the edges.
EdgePartition(int[], int[], Object, GraphXPrimitiveKeyOpenHashMap<Object, Object>, GraphXPrimitiveKeyOpenHashMap<Object, Object>, long[], Object, Option<OpenHashSet<Object>>, ClassTag<ED>, ClassTag<VD>) - Constructor for class org.apache.spark.graphx.impl.EdgePartition
 
EdgePartitionBuilder<ED,VD> - Class in org.apache.spark.graphx.impl
Constructs an EdgePartition from scratch.
EdgePartitionBuilder(int, ClassTag<ED>, ClassTag<VD>) - Constructor for class org.apache.spark.graphx.impl.EdgePartitionBuilder
 
edgePartitionToMsgs(int, EdgePartition<?, ?>) - Static method in class org.apache.spark.graphx.impl.RoutingTablePartition
Generate a `RoutingTableMessage` for each vertex referenced in `edgePartition`.
EdgeRDD<ED> - Class in org.apache.spark.graphx
EdgeRDD[ED, VD] extends RDD[Edge[ED} by storing the edges in columnar format on each partition for performance.
EdgeRDD(SparkContext, Seq<Dependency<?>>) - Constructor for class org.apache.spark.graphx.EdgeRDD
 
EdgeRDDImpl<ED,VD> - Class in org.apache.spark.graphx.impl
 
EdgeRDDImpl(RDD<Tuple2<Object, EdgePartition<ED, VD>>>, StorageLevel, ClassTag<ED>, ClassTag<VD>) - Constructor for class org.apache.spark.graphx.impl.EdgeRDDImpl
 
edges() - Method in class org.apache.spark.graphx.Graph
An RDD containing the edges and their associated attributes.
edges() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
edges() - Method in class org.apache.spark.graphx.impl.ReplicatedVertexView
 
EdgeTriplet<VD,ED> - Class in org.apache.spark.graphx
An edge triplet represents an edge along with the vertex attributes of its neighboring vertices.
EdgeTriplet() - Constructor for class org.apache.spark.graphx.EdgeTriplet
 
EdgeWithLocalIds<ED> - Class in org.apache.spark.graphx.impl
Add a new edge to the partition.
EdgeWithLocalIds(long, long, int, int, ED) - Constructor for class org.apache.spark.graphx.impl.EdgeWithLocalIds
 
EigenValueDecomposition - Class in org.apache.spark.mllib.linalg
:: Experimental :: Compute eigen-decomposition.
EigenValueDecomposition() - Constructor for class org.apache.spark.mllib.linalg.EigenValueDecomposition
 
Either() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges originating from *or* arriving at a vertex of interest.
elementClassTag() - Method in class org.apache.spark.rdd.RDD
 
elementIds() - Method in class org.apache.spark.mllib.recommendation.InLinkBlock
 
elementIds() - Method in class org.apache.spark.mllib.recommendation.OutLinkBlock
 
elements() - Method in class org.apache.spark.util.Vector
 
elementType() - Method in class org.apache.spark.sql.parquet.CatalystArrayContainsNullConverter
 
elementType() - Method in class org.apache.spark.sql.parquet.CatalystArrayConverter
 
elementType() - Method in class org.apache.spark.sql.parquet.CatalystNativeArrayConverter
 
emittedTaskSizeWarning() - Method in class org.apache.spark.scheduler.TaskSetManager
 
empty() - Static method in class org.apache.spark.graphx.impl.RoutingTablePartition
 
empty() - Static method in class org.apache.spark.ml.param.ParamMap
Returns an empty param map.
empty() - Static method in class org.apache.spark.scheduler.EventLoggingInfo
 
empty() - Static method in class org.apache.spark.storage.BlockStatus
 
empty() - Method in class org.apache.spark.util.TimeStampedHashMap
 
empty() - Method in class org.apache.spark.util.TimeStampedHashSet
 
empty() - Method in class org.apache.spark.util.TimeStampedWeakValueHashMap
 
emptyJson() - Static method in class org.apache.spark.util.Utils
Return an empty JSON object
emptyNode(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return a node with the given node id (but nothing else set).
emptyRDD() - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD that has no partitions or elements.
EmptyRDD<T> - Class in org.apache.spark.rdd
An RDD that has no partitions and no elements.
EmptyRDD(SparkContext, ClassTag<T>) - Constructor for class org.apache.spark.rdd.EmptyRDD
 
emptyRDD(ClassTag<T>) - Method in class org.apache.spark.SparkContext
Get an RDD that has no partitions or elements.
enableLogForwarding() - Static method in class org.apache.spark.sql.parquet.ParquetRelation
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.BooleanBitSet
 
encoder(NativeColumnType<T>) - Method in interface org.apache.spark.sql.columnar.compression.CompressionScheme
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.DictionaryEncoding
 
Encoder<T extends org.apache.spark.sql.catalyst.types.NativeType> - Interface in org.apache.spark.sql.columnar.compression
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.IntDelta
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.LongDelta
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.PassThrough
 
encoder(NativeColumnType<T>) - Static method in class org.apache.spark.sql.columnar.compression.RunLengthEncoding
 
end() - Method in class org.apache.spark.sql.parquet.CatalystArrayContainsNullConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystArrayConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystGroupConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystMapConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystNativeArrayConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystPrimitiveRowConverter
 
end() - Method in class org.apache.spark.sql.parquet.CatalystStructConverter
 
endIdx() - Method in class org.apache.spark.util.Distribution
 
endTime() - Method in class org.apache.spark.scheduler.ApplicationEventListener
 
endTime() - Method in class org.apache.spark.streaming.Interval
 
endTime() - Method in class org.apache.spark.streaming.util.WriteAheadLogManager.LogInfo
 
endTime() - Method in class org.apache.spark.ui.jobs.UIData.JobUIData
 
enforceCorrectType(Object, DataType) - Static method in class org.apache.spark.sql.json.JsonRDD
 
enqueueFailedTask(TaskSetManager, long, Enumeration.Value, ByteBuffer) - Method in class org.apache.spark.scheduler.TaskResultGetter
 
enqueueSuccessfulTask(TaskSetManager, long, ByteBuffer) - Method in class org.apache.spark.scheduler.TaskResultGetter
 
EnsembleCombiningStrategy - Class in org.apache.spark.mllib.tree.configuration
Enum to select ensemble combining strategy for base learners
EnsembleCombiningStrategy() - Constructor for class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
entries() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
 
Entropy - Class in org.apache.spark.mllib.tree.impurity
:: Experimental :: Class for calculating entropy during binary classification.
Entropy() - Constructor for class org.apache.spark.mllib.tree.impurity.Entropy
 
EntropyAggregator - Class in org.apache.spark.mllib.tree.impurity
Class for updating views of a vector of sufficient statistics, in order to compute impurity from a sample.
EntropyAggregator(int) - Constructor for class org.apache.spark.mllib.tree.impurity.EntropyAggregator
 
EntropyCalculator - Class in org.apache.spark.mllib.tree.impurity
Stores statistics for one (node, feature, bin) for calculating impurity.
EntropyCalculator(double[]) - Constructor for class org.apache.spark.mllib.tree.impurity.EntropyCalculator
 
entrySet() - Method in class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
env() - Method in class org.apache.spark.api.java.JavaSparkContext
 
env() - Method in class org.apache.spark.scheduler.TaskSetManager
 
env() - Method in class org.apache.spark.SparkContext
 
env() - Method in class org.apache.spark.streaming.scheduler.ReceiverTracker.ReceiverLauncher
 
env() - Method in class org.apache.spark.streaming.StreamingContext
 
environmentDetails() - Method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
environmentDetails(SparkConf, String, Seq<String>, Seq<String>) - Static method in class org.apache.spark.SparkEnv
Return a map representation of jvm information, Spark properties, system properties, and class paths.
EnvironmentListener - Class in org.apache.spark.ui.env
:: DeveloperApi :: A SparkListener that prepares information to be displayed on the EnvironmentTab
EnvironmentListener() - Constructor for class org.apache.spark.ui.env.EnvironmentListener
 
environmentListener() - Method in class org.apache.spark.ui.SparkUI
 
EnvironmentPage - Class in org.apache.spark.ui.env
 
EnvironmentPage(EnvironmentTab) - Constructor for class org.apache.spark.ui.env.EnvironmentPage
 
EnvironmentTab - Class in org.apache.spark.ui.env
 
EnvironmentTab(SparkUI) - Constructor for class org.apache.spark.ui.env.EnvironmentTab
 
environmentUpdateFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
environmentUpdateToJson(SparkListenerEnvironmentUpdate) - Static method in class org.apache.spark.util.JsonProtocol
 
envVars() - Method in class org.apache.spark.sql.execution.PythonUDF
 
epoch() - Method in class org.apache.spark.scheduler.Task
 
epoch() - Method in class org.apache.spark.scheduler.TaskSetManager
 
EPSILON() - Static method in class org.apache.spark.mllib.util.MLUtils
 
equals(Object) - Method in class org.apache.spark.graphx.EdgeDirection
 
equals(Object) - Method in class org.apache.spark.HashPartitioner
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
equals(Object) - Method in interface org.apache.spark.mllib.linalg.Vector
 
equals(Object) - Method in class org.apache.spark.mllib.recommendation.ALSPartitioner
 
equals(Object) - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
equals(Object) - Method in class org.apache.spark.RangePartitioner
 
equals(Object) - Method in class org.apache.spark.rdd.ParallelCollectionPartition
 
equals(Object) - Method in class org.apache.spark.scheduler.AccumulableInfo
 
equals(Object) - Method in class org.apache.spark.scheduler.InputFormatInfo
 
equals(Object) - Method in class org.apache.spark.scheduler.SplitInfo
 
equals(Object) - Method in class org.apache.spark.scheduler.Stage
 
equals(Object) - Method in class org.apache.spark.sql.api.java.ArrayType
 
equals(Object) - Method in class org.apache.spark.sql.api.java.DecimalType
 
equals(Object) - Method in class org.apache.spark.sql.api.java.MapType
 
equals(Object) - Method in class org.apache.spark.sql.api.java.Row
 
equals(Object) - Method in class org.apache.spark.sql.api.java.StructField
 
equals(Object) - Method in class org.apache.spark.sql.api.java.StructType
 
equals(Object) - Method in class org.apache.spark.sql.api.java.UserDefinedType
 
equals(Object) - Method in class org.apache.spark.sql.parquet.ParquetRelation
 
equals(Object) - Method in class org.apache.spark.sql.sources.LogicalRelation
 
equals(Object) - Method in class org.apache.spark.storage.BlockId
 
equals(Object) - Method in class org.apache.spark.storage.BlockManagerId
 
equals(Object) - Method in class org.apache.spark.storage.StorageLevel
 
EqualTo - Class in org.apache.spark.sql.sources
 
EqualTo(String, Object) - Constructor for class org.apache.spark.sql.sources.EqualTo
 
error(SchedulerDriver, String) - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
error(SchedulerDriver, String) - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
error(String) - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
error() - Method in class org.apache.spark.streaming.scheduler.DeregisterReceiver
 
error() - Method in class org.apache.spark.streaming.scheduler.ReportError
 
errorMessage() - Method in class org.apache.spark.ui.jobs.UIData.TaskUIData
 
ErrorReported - Class in org.apache.spark.streaming.scheduler
 
ErrorReported(String, Throwable) - Constructor for class org.apache.spark.streaming.scheduler.ErrorReported
 
estimate(Object) - Static method in class org.apache.spark.util.SizeEstimator
 
Estimator<M extends Model<M>> - Class in org.apache.spark.ml
:: AlphaComponent :: Abstract class for estimators that fit models to data.
Estimator() - Constructor for class org.apache.spark.ml.Estimator
 
estimator() - Method in interface org.apache.spark.ml.tuning.CrossValidatorParams
param for the estimator to be cross-validated
estimatorParamMaps() - Method in interface org.apache.spark.ml.tuning.CrossValidatorParams
param for estimator param maps
eval(Row) - Method in class org.apache.spark.sql.execution.PythonUDF
 
eval(Row) - Method in class org.apache.spark.sql.hive.HiveGenericUdf
 
eval(Row) - Method in class org.apache.spark.sql.hive.HiveGenericUdtf
 
eval(Row) - Method in class org.apache.spark.sql.hive.HiveSimpleUdf
 
eval(Row) - Method in class org.apache.spark.sql.hive.HiveUdafFunction
 
evaluate(SchemaRDD, ParamMap) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
evaluate(SchemaRDD, ParamMap) - Method in class org.apache.spark.ml.Evaluator
Evaluates the output.
EvaluatePython - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Evaluates a PythonUDF, appending the result to the end of the input tuple.
EvaluatePython(PythonUDF, LogicalPlan, AttributeReference) - Constructor for class org.apache.spark.sql.execution.EvaluatePython
 
Evaluator - Class in org.apache.spark.ml
:: AlphaComponent :: Abstract class for evaluators that compute metrics from predictions.
Evaluator() - Constructor for class org.apache.spark.ml.Evaluator
 
evaluator() - Method in interface org.apache.spark.ml.tuning.CrossValidatorParams
param for the evaluator for selection
event() - Method in class org.apache.spark.streaming.flume.SparkFlumeEvent
 
eventLogDir() - Method in class org.apache.spark.SparkContext
 
eventLogger() - Method in class org.apache.spark.SparkContext
 
EventLoggingInfo - Class in org.apache.spark.scheduler
Information needed to process the event logs associated with an application.
EventLoggingInfo(Seq<Path>, String, Option<CompressionCodec>, boolean) - Constructor for class org.apache.spark.scheduler.EventLoggingInfo
 
EventLoggingListener - Class in org.apache.spark.scheduler
A SparkListener that logs events to persistent storage.
EventLoggingListener(String, String, SparkConf, Configuration) - Constructor for class org.apache.spark.scheduler.EventLoggingListener
 
EventLoggingListener(String, String, SparkConf) - Constructor for class org.apache.spark.scheduler.EventLoggingListener
 
eventProcessActor() - Method in class org.apache.spark.scheduler.DAGScheduler
 
EventTransformer - Class in org.apache.spark.streaming.flume
A simple object that provides the implementation of readExternal and writeExternal for both the wrapper classes for Flume-style Events.
EventTransformer() - Constructor for class org.apache.spark.streaming.flume.EventTransformer
 
ExamplePoint - Class in org.apache.spark.sql.test
An example class to demonstrate UDT in Scala, Java, and Python.
ExamplePoint(double, double) - Constructor for class org.apache.spark.sql.test.ExamplePoint
 
ExamplePointUDT - Class in org.apache.spark.sql.test
User-defined type for ExamplePoint.
ExamplePointUDT() - Constructor for class org.apache.spark.sql.test.ExamplePointUDT
 
Except - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Returns a table with the elements from left that are not in right using the built-in spark subtract function.
Except(SparkPlan, SparkPlan) - Constructor for class org.apache.spark.sql.execution.Except
 
except(SchemaRDD) - Method in class org.apache.spark.sql.SchemaRDD
Performs a relational except on two SchemaRDDs
exception() - Method in class org.apache.spark.scheduler.JobFailed
 
EXCEPTION_PRINT_INTERVAL() - Method in class org.apache.spark.scheduler.TaskSetManager
 
ExceptionFailure - Class in org.apache.spark
:: DeveloperApi :: Task failed due to a runtime exception.
ExceptionFailure(String, String, StackTraceElement[], String, Option<TaskMetrics>) - Constructor for class org.apache.spark.ExceptionFailure
 
ExceptionFailure(Throwable, Option<TaskMetrics>) - Constructor for class org.apache.spark.ExceptionFailure
 
exceptionFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
exceptionString(Throwable) - Static method in class org.apache.spark.util.Utils
Return a nice string representation of the exception.
exceptionToJson(Exception) - Static method in class org.apache.spark.util.JsonProtocol
 
Exchange - Class in org.apache.spark.sql.execution
:: DeveloperApi ::
Exchange(Partitioning, SparkPlan) - Constructor for class org.apache.spark.sql.execution.Exchange
 
execArgs() - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
execId() - Method in class org.apache.spark.ExecutorLostFailure
 
execId() - Method in class org.apache.spark.scheduler.ExecutorAdded
 
execId() - Method in class org.apache.spark.scheduler.ExecutorLost
 
execId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
execId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor
 
execute() - Method in class org.apache.spark.sql.columnar.InMemoryColumnarTableScan
 
execute() - Method in class org.apache.spark.sql.execution.Aggregate
Substituted version of aggregateExpressions expressions which are used to compute final output rows given a group and the result of all aggregate computations.
execute() - Method in class org.apache.spark.sql.execution.BatchPythonEvaluation
 
execute() - Method in interface org.apache.spark.sql.execution.Command
 
execute() - Method in class org.apache.spark.sql.execution.Distinct
 
execute() - Method in class org.apache.spark.sql.execution.Except
 
execute() - Method in class org.apache.spark.sql.execution.Exchange
 
execute() - Method in class org.apache.spark.sql.execution.ExecutedCommand
 
execute() - Method in class org.apache.spark.sql.execution.ExistingRdd
 
execute() - Method in class org.apache.spark.sql.execution.ExternalSort
 
execute() - Method in class org.apache.spark.sql.execution.Filter
 
execute() - Method in class org.apache.spark.sql.execution.Generate
 
execute() - Method in class org.apache.spark.sql.execution.GeneratedAggregate
 
execute() - Method in class org.apache.spark.sql.execution.Intersect
 
execute() - Method in class org.apache.spark.sql.execution.joins.BroadcastHashJoin
 
execute() - Method in class org.apache.spark.sql.execution.joins.BroadcastNestedLoopJoin
 
execute() - Method in class org.apache.spark.sql.execution.joins.CartesianProduct
 
execute() - Method in class org.apache.spark.sql.execution.joins.HashOuterJoin
 
execute() - Method in class org.apache.spark.sql.execution.joins.LeftSemiJoinBNL
 
execute() - Method in class org.apache.spark.sql.execution.joins.LeftSemiJoinHash
 
execute() - Method in class org.apache.spark.sql.execution.joins.ShuffledHashJoin
 
execute() - Method in class org.apache.spark.sql.execution.Limit
 
execute() - Method in class org.apache.spark.sql.execution.OutputFaker
 
execute() - Method in class org.apache.spark.sql.execution.PhysicalRDD
 
execute() - Method in class org.apache.spark.sql.execution.Project
 
execute() - Method in class org.apache.spark.sql.execution.Sample
 
execute() - Method in class org.apache.spark.sql.execution.Sort
 
execute() - Method in class org.apache.spark.sql.execution.SparkPlan
Runs this query returning the result as an RDD.
execute() - Method in class org.apache.spark.sql.execution.TakeOrdered
 
execute() - Method in class org.apache.spark.sql.execution.Union
 
execute() - Method in class org.apache.spark.sql.hive.execution.CreateTableAsSelect
 
execute() - Method in class org.apache.spark.sql.hive.execution.HiveTableScan
 
execute() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformation
 
execute() - Method in class org.apache.spark.sql.parquet.InsertIntoParquetTable
Inserts all rows into the Parquet file.
execute() - Method in class org.apache.spark.sql.parquet.ParquetTableScan
 
execute(Seq<String>, File) - Static method in class org.apache.spark.util.Utils
Execute a command in the given working directory, throwing an exception if it completes with an exit code other than 0.
executeAndGetOutput(Seq<String>, File, Map<String, String>) - Static method in class org.apache.spark.util.Utils
Execute a command and get its output, throwing an exception if it yields a code other than 0.
executeCollect() - Method in interface org.apache.spark.sql.execution.Command
 
executeCollect() - Method in class org.apache.spark.sql.execution.ExecutedCommand
 
executeCollect() - Method in class org.apache.spark.sql.execution.Limit
A custom implementation modeled after the take function on RDDs but which never runs any job locally.
executeCollect() - Method in class org.apache.spark.sql.execution.SparkPlan
Runs this query returning the result as an array.
executeCollect() - Method in class org.apache.spark.sql.execution.TakeOrdered
 
ExecutedCommand - Class in org.apache.spark.sql.execution
 
ExecutedCommand(RunnableCommand) - Constructor for class org.apache.spark.sql.execution.ExecutedCommand
 
executePlan(LogicalPlan) - Method in class org.apache.spark.sql.hive.test.TestHiveContext
 
executor() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
executor() - Method in class org.apache.spark.scheduler.local.LocalActor
 
executor() - Method in class org.apache.spark.streaming.CheckpointWriter
 
executor_() - Method in class org.apache.spark.streaming.receiver.Receiver
Handler object that runs the receiver.
executorActor() - Method in class org.apache.spark.scheduler.cluster.ExecutorData
 
executorActorSystemName() - Static method in class org.apache.spark.SparkEnv
 
executorAdded(String, String, String, int, int) - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
executorAdded(String, String) - Method in class org.apache.spark.scheduler.DAGScheduler
 
ExecutorAdded - Class in org.apache.spark.scheduler
 
ExecutorAdded(String, String) - Constructor for class org.apache.spark.scheduler.ExecutorAdded
 
executorAdded(String, String) - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
executorAdded() - Method in class org.apache.spark.scheduler.TaskSetManager
 
executorAddress() - Method in class org.apache.spark.scheduler.cluster.ExecutorData
 
ExecutorAllocationManager - Class in org.apache.spark
An agent that dynamically allocates and removes executors based on the workload.
ExecutorAllocationManager(SparkContext) - Constructor for class org.apache.spark.ExecutorAllocationManager
 
executorAllocationManager() - Method in class org.apache.spark.SparkContext
 
ExecutorCacheTaskLocation - Class in org.apache.spark.scheduler
A location that includes both a host and an executor id on that host.
ExecutorCacheTaskLocation(String, String) - Constructor for class org.apache.spark.scheduler.ExecutorCacheTaskLocation
 
ExecutorData - Class in org.apache.spark.scheduler.cluster
Grouping of data for an executor used by CoarseGrainedSchedulerBackend.
ExecutorData(ActorRef, Address, String, int, int) - Constructor for class org.apache.spark.scheduler.cluster.ExecutorData
 
executorEnvs() - Method in class org.apache.spark.SparkContext
 
ExecutorExited - Class in org.apache.spark.scheduler
 
ExecutorExited(int) - Constructor for class org.apache.spark.scheduler.ExecutorExited
 
executorHeartbeatReceived(String, Tuple4<Object, Object, Object, TaskMetrics>[], BlockManagerId) - Method in class org.apache.spark.scheduler.DAGScheduler
Update metrics for in-progress tasks and let the master know that the BlockManager is still alive.
executorHeartbeatReceived(String, Tuple2<Object, TaskMetrics>[], BlockManagerId) - Method in interface org.apache.spark.scheduler.TaskScheduler
Update metrics for in-progress tasks and let the master know that the BlockManager is still alive.
executorHeartbeatReceived(String, Tuple2<Object, TaskMetrics>[], BlockManagerId) - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
Update metrics for in-progress tasks and let the master know that the BlockManager is still alive.
executorHost() - Method in class org.apache.spark.scheduler.cluster.ExecutorData
 
executorId() - Method in class org.apache.spark.Heartbeat
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
executorId() - Method in class org.apache.spark.scheduler.ExecutorCacheTaskLocation
 
executorId() - Method in class org.apache.spark.scheduler.TaskDescription
 
executorId() - Method in class org.apache.spark.scheduler.TaskInfo
 
executorId() - Method in class org.apache.spark.scheduler.WorkerOffer
 
executorId() - Method in class org.apache.spark.SparkEnv
 
executorId() - Method in class org.apache.spark.storage.BlockManagerId
 
executorId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetActorSystemHostPortForExecutor
 
executorIds() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors
 
executorIdToBlockManagerId() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
executorIdToStorageStatus() - Method in class org.apache.spark.storage.StorageStatusListener
 
ExecutorLossReason - Class in org.apache.spark.scheduler
Represents an explanation for a executor or whole slave failing or exiting.
ExecutorLossReason(String) - Constructor for class org.apache.spark.scheduler.ExecutorLossReason
 
executorLost(SchedulerDriver, Protos.ExecutorID, Protos.SlaveID, int) - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
executorLost(SchedulerDriver, Protos.ExecutorID, Protos.SlaveID, int) - Method in class org.apache.spark.scheduler.cluster.mesos.MesosSchedulerBackend
 
executorLost(String) - Method in class org.apache.spark.scheduler.DAGScheduler
 
ExecutorLost - Class in org.apache.spark.scheduler
 
ExecutorLost(String) - Constructor for class org.apache.spark.scheduler.ExecutorLost
 
executorLost(String, String) - Method in class org.apache.spark.scheduler.Pool
 
executorLost(String, String) - Method in interface org.apache.spark.scheduler.Schedulable
 
executorLost(String, ExecutorLossReason) - Method in class org.apache.spark.scheduler.TaskSchedulerImpl
 
executorLost(String, String) - Method in class org.apache.spark.scheduler.TaskSetManager
Called by TaskScheduler when an executor is lost so we can re-enqueue our tasks
ExecutorLostFailure - Class in org.apache.spark
:: DeveloperApi :: The task failed because the executor that it was running on was lost.
ExecutorLostFailure(String) - Constructor for class org.apache.spark.ExecutorLostFailure
 
executorMemory() - Method in class org.apache.spark.SparkContext
 
executorPct() - Method in class org.apache.spark.scheduler.RuntimePercentage
 
executorRemoved(String, String, Option<Object>) - Method in class org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend
 
executorRunTime() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
executorSideSetup(int, int, int) - Method in class org.apache.spark.sql.hive.SparkHiveWriterContainer
 
ExecutorsListener - Class in org.apache.spark.ui.exec
:: DeveloperApi :: A SparkListener that prepares information to be displayed on the ExecutorsTab
ExecutorsListener(StorageStatusListener) - Constructor for class org.apache.spark.ui.exec.ExecutorsListener
 
executorsListener() - Method in class org.apache.spark.ui.SparkUI
 
ExecutorsPage - Class in org.apache.spark.ui.exec
 
ExecutorsPage(ExecutorsTab, boolean) - Constructor for class org.apache.spark.ui.exec.ExecutorsPage
 
ExecutorsTab - Class in org.apache.spark.ui.exec
 
ExecutorsTab(SparkUI) - Constructor for class org.apache.spark.ui.exec.ExecutorsTab
 
executorSummary() - Method in class org.apache.spark.ui.jobs.UIData.StageUIData
 
ExecutorSummaryInfo - Class in org.apache.spark.ui.exec
Summary information about an executor to display in the UI.
ExecutorSummaryInfo(String, String, int, long, long, int, int, int, int, long, long, long, long, long) - Constructor for class org.apache.spark.ui.exec.ExecutorSummaryInfo
 
ExecutorTable - Class in org.apache.spark.ui.jobs
Stage summary grouped by executors.
ExecutorTable(int, int, StagesTab) - Constructor for class org.apache.spark.ui.jobs.ExecutorTable
 
ExecutorThreadDumpPage - Class in org.apache.spark.ui.exec
 
ExecutorThreadDumpPage(ExecutorsTab) - Constructor for class org.apache.spark.ui.exec.ExecutorThreadDumpPage
 
executorToDuration() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToInputBytes() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToOutputBytes() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToShuffleRead() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToShuffleWrite() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToTasksActive() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToTasksComplete() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
executorToTasksFailed() - Method in class org.apache.spark.ui.exec.ExecutorsListener
 
ExistingEdgePartitionBuilder<ED,VD> - Class in org.apache.spark.graphx.impl
Constructs an EdgePartition from an existing EdgePartition with the same vertex set.
ExistingEdgePartitionBuilder(GraphXPrimitiveKeyOpenHashMap<Object, Object>, long[], Object, Option<OpenHashSet<Object>>, int, ClassTag<ED>, ClassTag<VD>) - Constructor for class org.apache.spark.graphx.impl.ExistingEdgePartitionBuilder
 
ExistingRdd - Class in org.apache.spark.sql.execution
 
ExistingRdd(Seq<Attribute>, RDD<Row>) - Constructor for class org.apache.spark.sql.execution.ExistingRdd
 
exitCode() - Method in class org.apache.spark.scheduler.ExecutorExited
 
Experimental - Annotation Type in org.apache.spark.annotation
An experimental user-facing API.
ExplainCommand - Class in org.apache.spark.sql.execution
An explain command for users to see how a command will be executed.
ExplainCommand(LogicalPlan, Seq<Attribute>, boolean, SQLContext) - Constructor for class org.apache.spark.sql.execution.ExplainCommand
 
explainedVariance() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns the explained variance regression score.
explainParams() - Method in interface org.apache.spark.ml.param.Params
Returns the documentation of all params.
explode() - Static method in class org.apache.spark.sql.hive.HiveQl
 
exprs() - Method in class org.apache.spark.sql.hive.HiveUdafFunction
 
extended() - Method in class org.apache.spark.sql.execution.ExplainCommand
 
ExtendedHiveQlParser - Class in org.apache.spark.sql.hive
A parser that recognizes all HiveQL constructs together with Spark SQL specific extensions.
ExtendedHiveQlParser() - Constructor for class org.apache.spark.sql.hive.ExtendedHiveQlParser
 
externalShuffleServiceEnabled() - Method in class org.apache.spark.storage.BlockManager
 
ExternalSort - Class in org.apache.spark.sql.execution
:: DeveloperApi :: Performs a sort, spilling to disk as needed.
ExternalSort(Seq<SortOrder>, boolean, SparkPlan) - Constructor for class org.apache.spark.sql.execution.ExternalSort
 
externalSortEnabled() - Method in interface org.apache.spark.sql.SQLConf
When true the planner will use the external sort, which may spill to disk.
extraCoresPerSlave() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BOOLEAN
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.BOOLEAN
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.BYTE
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.BYTE
 
extract(ByteBuffer) - Method in class org.apache.spark.sql.columnar.ByteArrayColumnType
 
extract(ByteBuffer) - Method in class org.apache.spark.sql.columnar.ColumnType
Extracts a value out of the buffer at the buffer's current position.
extract(ByteBuffer, MutableRow, int) - Method in class org.apache.spark.sql.columnar.ColumnType
Extracts a value out of the buffer at the buffer's current position and stores in row(ordinal).
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.DATE
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.DOUBLE
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.DOUBLE
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.FLOAT
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.FLOAT
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.INT
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.INT
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.LONG
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.LONG
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.SHORT
 
extract(ByteBuffer, MutableRow, int) - Static method in class org.apache.spark.sql.columnar.SHORT
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.STRING
 
extract(ByteBuffer) - Static method in class org.apache.spark.sql.columnar.TIMESTAMP
 
extractDistribution(Function1<BatchInfo, Option<Object>>) - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
extractDoubleDistribution(Seq<Tuple2<TaskInfo, TaskMetrics>>, Function2<TaskInfo, TaskMetrics, Option<Object>>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
extractFn() - Method in class org.apache.spark.ui.JettyUtils.ServletParams
 
extractLongDistribution(Seq<Tuple2<TaskInfo, TaskMetrics>>, Function2<TaskInfo, TaskMetrics, Option<Object>>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
extractMultiClassCategories(int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Nested method to extract list of eligible categories given an index.
ExtractPythonUdfs - Class in org.apache.spark.sql.execution
Extracts PythonUDFs from operators, rewriting the query plan so that the UDF can be evaluated alone in a batch.
ExtractPythonUdfs() - Constructor for class org.apache.spark.sql.execution.ExtractPythonUdfs
 
extractSingle(MutableRow, int) - Method in class org.apache.spark.sql.columnar.BasicColumnAccessor
 
extractSingle(MutableRow, int) - Method in interface org.apache.spark.sql.columnar.compression.CompressibleColumnAccessor
 
extractTo(MutableRow, int) - Method in class org.apache.spark.sql.columnar.BasicColumnAccessor
 
extractTo(MutableRow, int) - Method in interface org.apache.spark.sql.columnar.ColumnAccessor
 
extractTo(MutableRow, int) - Method in interface org.apache.spark.sql.columnar.NullableColumnAccessor
 
extraStrategies() - Method in class org.apache.spark.sql.SQLContext
:: DeveloperApi :: Allows extra strategies to be injected into the query planner at runtime.
eye(int) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate an Identity Matrix in DenseMatrix format.

F

f() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD2
 
f() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD3
 
f() - Method in class org.apache.spark.rdd.ZippedPartitionsRDD4
 
f1Measure() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns document-based f1-measure averaged by the number of documents
f1Measure(double) - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns f1-measure for a given label (category)
failed() - Method in class org.apache.spark.scheduler.TaskInfo
 
FAILED() - Static method in class org.apache.spark.TaskState
 
failedJobs() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
failedStages() - Method in class org.apache.spark.scheduler.DAGScheduler
 
failedStages() - Method in class org.apache.spark.ui.jobs.JobProgressListener
 
FailedStageTable - Class in org.apache.spark.ui.jobs
 
FailedStageTable(Seq<StageInfo>, String, JobProgressListener, boolean) - Constructor for class org.apache.spark.ui.jobs.FailedStageTable
 
failedTasks() - Method in class org.apache.spark.ui.exec.ExecutorSummaryInfo
 
failedTasks() - Method in class org.apache.spark.ui.jobs.UIData.ExecutorSummary
 
failure() - Method in class org.apache.spark.partial.ApproximateActionListener
 
failureReason() - Method in class org.apache.spark.scheduler.StageInfo
If the stage failed, the reason why.
failuresBySlaveId() - Method in class org.apache.spark.scheduler.cluster.mesos.CoarseMesosSchedulerBackend
 
FAIR() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
FAIR_SCHEDULER_PROPERTIES() - Method in class org.apache.spark.scheduler.FairSchedulableBuilder
 
FairSchedulableBuilder - Class in org.apache.spark.scheduler
 
FairSchedulableBuilder(Pool, SparkConf) - Constructor for class org.apache.spark.scheduler.FairSchedulableBuilder
 
FairSchedulingAlgorithm - Class in org.apache.spark.scheduler
 
FairSchedulingAlgorithm() - Constructor for class org.apache.spark.scheduler.FairSchedulingAlgorithm
 
fakeClassTag() - Static method in class org.apache.spark.api.java.JavaSparkContext
Produces a ClassTag[T], which is actually just a casted ClassTag[AnyRef].
fakeOutput(Seq<Attribute>) - Method in class org.apache.spark.sql.hive.HiveStrategies.ParquetConversion.PhysicalPlanHacks
 
FakeParquetSerDe - Class in org.apache.spark.sql.hive.parquet
A placeholder that allows Spark SQL users to create metastore tables that are stored as parquet files.
FakeParquetSerDe() - Constructor for class org.apache.spark.sql.hive.parquet.FakeParquetSerDe
 
FALSE() - Static method in class org.apache.spark.sql.hive.HiveQl
 
FalsePositiveRate - Class in org.apache.spark.mllib.evaluation.binary
False positive rate.
FalsePositiveRate() - Constructor for class org.apache.spark.mllib.evaluation.binary.