Packages
Package
Description
 
Core Spark classes in Scala.
Spark Java programming APIs.
Set of interfaces to represent functions in Spark's Java API.
 
 
 
Spark's broadcast variables, used to broadcast immutable datasets to all nodes.
 
ALPHA COMPONENT GraphX is a graph processing framework built on top of Spark.
 
Various analytics functions for graphs.
Collections of utilities used by graphx.
 
IO codecs used for compression.
 
Library for launching Spark applications programmatically.
 
 
 
 
DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.
 
ML attributes
 
 
 
Feature transformers The `ml.feature` package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
RDD-based machine learning APIs (in maintenance mode).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Provides implementation's of various RDDs.
 
Spark's DAG scheduler.
 
 
 
Pluggable serializers for RDD and shuffle data.
 
 
 
 
Allows the execution of relational queries, including those expressed in SQL using Spark.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Java APIs for spark streaming.
Various implementations of DStreams.
 
 
 
 
 
 
 
 
 
 
 
 
Spark utilities.
 
Utilities for random number generation.