Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
A name for your job, to display on the cluster web UI
The SPARK_HOME directory on the slave nodes
Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
Environment variables to set on worker nodes
Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
A name for your job, to display on the cluster web UI
The SPARK_HOME directory on the slave nodes
Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.
Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
A name for your job, to display on the cluster web UI
The SPARK_HOME directory on the slave nodes
Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).
A name for your job, to display on the cluster web UI
Create an Accumulator variable of a given type, which tasks can "add" values
to using the +=
method.
Create an Accumulator variable of a given type, which tasks can "add" values
to using the +=
method. Only the master can access the accumulator's value
.
Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for reading it in distributed functions.
Broadcast a read-only variable to the cluster, returning a spark.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.
Create an Accumulator double variable, which tasks can "add" values
to using the +=
method.
Create an Accumulator double variable, which tasks can "add" values
to using the +=
method. Only the master can access the accumulator's value
.
Get Spark's home location from either a value set through the constructor, or the spark.
Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.
Get an RDD for a Hadoop file with an arbitrary InputFormat
Get an RDD for a Hadoop file with an arbitrary InputFormat
Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.
Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).
Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.
Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).
Create an Accumulator integer variable, which tasks can "add" values
to using the +=
method.
Create an Accumulator integer variable, which tasks can "add" values
to using the +=
method. Only the master can access the accumulator's value
.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Distribute a local Scala collection to form an RDD.
Get an RDD for a Hadoop SequenceFile.
Get an RDD for a Hadoop SequenceFile with given key and value types.
Shut down the SparkContext.
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
Build the union of two or more RDDs.
Build the union of two or more RDDs.
Build the union of two or more RDDs.
Build the union of two or more RDDs.
Build the union of two or more RDDs.
Build the union of two or more RDDs.
A Java-friendly version of SparkContext that returns JavaRDDs and works with Java collections instead of Scala ones.