org.apache.spark.api.java

JavaSparkContext

class JavaSparkContext extends JavaSparkContextVarargsWorkaround

A Java-friendly version of org.apache.spark.SparkContext that returns org.apache.spark.api.java.JavaRDDs and works with Java collections instead of Scala ones.

Linear Supertypes
JavaSparkContextVarargsWorkaround, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. JavaSparkContext
  2. JavaSparkContextVarargsWorkaround
  3. AnyRef
  4. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JavaSparkContext(master: String, appName: String, sparkHome: String, jars: Array[String], environment: Map[String, String])

    master

    Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).

    appName

    A name for your application, to display on the cluster web UI

    sparkHome

    The SPARK_HOME directory on the slave nodes

    jars

    Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.

    environment

    Environment variables to set on worker nodes

  2. new JavaSparkContext(master: String, appName: String, sparkHome: String, jars: Array[String])

    master

    Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).

    appName

    A name for your application, to display on the cluster web UI

    sparkHome

    The SPARK_HOME directory on the slave nodes

    jars

    Collection of JARs to send to the cluster. These can be paths on the local file system or HDFS, HTTP, HTTPS, or FTP URLs.

  3. new JavaSparkContext(master: String, appName: String, sparkHome: String, jarFile: String)

    master

    Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).

    appName

    A name for your application, to display on the cluster web UI

    sparkHome

    The SPARK_HOME directory on the slave nodes

    jarFile

    JAR file to send to the cluster. This can be a path on the local file system or an HDFS, HTTP, HTTPS, or FTP URL.

  4. new JavaSparkContext(master: String, appName: String, conf: SparkConf)

    master

    Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).

    appName

    A name for your application, to display on the cluster web UI

    conf

    a org.apache.spark.SparkConf object specifying other Spark parameters

  5. new JavaSparkContext(master: String, appName: String)

    master

    Cluster URL to connect to (e.g. mesos://host:port, spark://host:port, local[4]).

    appName

    A name for your application, to display on the cluster web UI

  6. new JavaSparkContext(conf: SparkConf)

    conf

    a org.apache.spark.SparkConf object specifying Spark parameters

  7. new JavaSparkContext(sc: SparkContext)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def accumulable[T, R](initialValue: T, param: AccumulableParam[T, R]): Accumulable[T, R]

    Create an org.apache.spark.Accumulable shared variable of the given type, to which tasks can "add" values with add.

    Create an org.apache.spark.Accumulable shared variable of the given type, to which tasks can "add" values with add. Only the master can access the accumuable's value.

  7. def accumulator[T](initialValue: T, accumulatorParam: AccumulatorParam[T]): Accumulator[T]

    Create an org.apache.spark.Accumulator variable of a given type, which tasks can "add" values to using the add method.

    Create an org.apache.spark.Accumulator variable of a given type, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

  8. def accumulator(initialValue: Double): Accumulator[Double]

    Create an org.apache.spark.Accumulator double variable, which tasks can "add" values to using the add method.

    Create an org.apache.spark.Accumulator double variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

  9. def accumulator(initialValue: Int): Accumulator[Integer]

    Create an org.apache.spark.Accumulator integer variable, which tasks can "add" values to using the add method.

    Create an org.apache.spark.Accumulator integer variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

  10. def addFile(path: String): Unit

    Add a file to be downloaded with this Spark job on every node.

    Add a file to be downloaded with this Spark job on every node. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI. To access the file in Spark jobs, use SparkFiles.get(path) to find its download location.

  11. def addJar(path: String): Unit

    Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

    Adds a JAR dependency for all tasks to be executed on this SparkContext in the future. The path passed can be either a local file, a file in HDFS (or other Hadoop-supported filesystems), or an HTTP, HTTPS or FTP URI.

  12. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  13. def broadcast[T](value: T): Broadcast[T]

    Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions.

    Broadcast a read-only variable to the cluster, returning a org.apache.spark.broadcast.Broadcast object for reading it in distributed functions. The variable will be sent to each cluster only once.

  14. def cancelAllJobs(): Unit

    Cancel all jobs that have been scheduled or are running.

  15. def cancelJobGroup(groupId: String): Unit

    Cancel active jobs for the specified group.

    Cancel active jobs for the specified group. See org.apache.spark.api.java.JavaSparkContext.setJobGroup for more information.

  16. def checkpointFile[T](path: String): JavaRDD[T]

    Attributes
    protected
  17. def clearCallSite(): Unit

    Pass-through to SparkContext.

    Pass-through to SparkContext.setCallSite. For API support only.

  18. def clearFiles(): Unit

    Clear the job's list of files added by addFile so that they do not get downloaded to any new nodes.

  19. def clearJars(): Unit

    Clear the job's list of JARs added by addJar so that they do not get downloaded to any new nodes.

  20. def clearJobGroup(): Unit

    Clear the current thread's job group ID and its description.

  21. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  22. def doubleAccumulator(initialValue: Double): Accumulator[Double]

    Create an org.apache.spark.Accumulator double variable, which tasks can "add" values to using the add method.

    Create an org.apache.spark.Accumulator double variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

  23. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  24. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  25. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. def getCheckpointDir: Optional[String]

  27. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  28. def getConf: SparkConf

    Return a copy of this JavaSparkContext's configuration.

    Return a copy of this JavaSparkContext's configuration. The configuration cannot be changed at runtime.

  29. def getLocalProperty(key: String): String

    Get a local property set in this thread, or null if it is missing.

    Get a local property set in this thread, or null if it is missing. See org.apache.spark.api.java.JavaSparkContext.setLocalProperty.

  30. def getSparkHome(): Optional[String]

    Get Spark's home location from either a value set through the constructor, or the spark.

    Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference). If neither of these is set, return None.

  31. def hadoopConfiguration(): Configuration

    Returns the Hadoop configuration used for the Hadoop code (e.

    Returns the Hadoop configuration used for the Hadoop code (e.g. file systems) we reuse.

  32. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat

    Get an RDD for a Hadoop file with an arbitrary InputFormat

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  33. def hadoopFile[K, V, F <: InputFormat[K, V]](path: String, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop file with an arbitrary InputFormat.

    Get an RDD for a Hadoop file with an arbitrary InputFormat.

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  34. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  35. def hadoopRDD[K, V, F <: InputFormat[K, V]](conf: JobConf, inputFormatClass: Class[F], keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.

    Get an RDD for a Hadoop-readable dataset from a Hadooop JobConf giving its InputFormat and any other necessary info (e.g. file name for a filesystem-based dataset, table name for HyperTable, etc).

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  36. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  37. def intAccumulator(initialValue: Int): Accumulator[Integer]

    Create an org.apache.spark.Accumulator integer variable, which tasks can "add" values to using the add method.

    Create an org.apache.spark.Accumulator integer variable, which tasks can "add" values to using the add method. Only the master can access the accumulator's value.

  38. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  39. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  40. def newAPIHadoopFile[K, V, F <: InputFormat[K, V]](path: String, fClass: Class[F], kClass: Class[K], vClass: Class[V], conf: Configuration): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  41. def newAPIHadoopRDD[K, V, F <: InputFormat[K, V]](conf: Configuration, fClass: Class[F], kClass: Class[K], vClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

    Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  42. final def notify(): Unit

    Definition Classes
    AnyRef
  43. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  44. def objectFile[T](path: String): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

  45. def objectFile[T](path: String, minSplits: Int): JavaRDD[T]

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.

    Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition. This is still an experimental storage format and may not be supported exactly as is in future Spark releases. It will also be pretty slow if you use the default serializer (Java serialization), though the nice thing about it is that there's very little effort required to save arbitrary objects.

  46. def parallelize[T](list: List[T]): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

  47. def parallelize[T](list: List[T], numSlices: Int): JavaRDD[T]

    Distribute a local Scala collection to form an RDD.

  48. def parallelizeDoubles(list: List[Double]): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

  49. def parallelizeDoubles(list: List[Double], numSlices: Int): JavaDoubleRDD

    Distribute a local Scala collection to form an RDD.

  50. def parallelizePairs[K, V](list: List[(K, V)]): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

  51. def parallelizePairs[K, V](list: List[(K, V)], numSlices: Int): JavaPairRDD[K, V]

    Distribute a local Scala collection to form an RDD.

  52. val sc: SparkContext

  53. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V]): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile.

    Get an RDD for a Hadoop SequenceFile.

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  54. def sequenceFile[K, V](path: String, keyClass: Class[K], valueClass: Class[V], minSplits: Int): JavaPairRDD[K, V]

    Get an RDD for a Hadoop SequenceFile with given key and value types.

    Get an RDD for a Hadoop SequenceFile with given key and value types.

    Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD will create many references to the same object. If you plan to directly cache Hadoop writable objects, you should first copy them using a map function.

  55. def setCallSite(site: String): Unit

    Pass-through to SparkContext.

    Pass-through to SparkContext.setCallSite. For API support only.

  56. def setCheckpointDir(dir: String): Unit

    Set the directory under which RDDs are going to be checkpointed.

    Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS path if running on a cluster.

  57. def setJobGroup(groupId: String, description: String): Unit

    Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

    Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.

    Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application programmers can use this method to group all those jobs together and give a group description. Once set, the Spark web UI will associate such jobs with this group.

    The application can also use org.apache.spark.api.java.JavaSparkContext.cancelJobGroup to cancel all running jobs in this group. For example,

    // In the main thread:
    sc.setJobGroup("some_job_to_cancel", "some job description");
    rdd.map(...).count();
    
    // In a separate thread:
    sc.cancelJobGroup("some_job_to_cancel");
  58. def setLocalProperty(key: String, value: String): Unit

    Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.

  59. def stop(): Unit

    Shut down the SparkContext.

  60. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  61. def textFile(path: String, minSplits: Int): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

  62. def textFile(path: String): JavaRDD[String]

    Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.

  63. def toString(): String

    Definition Classes
    AnyRef → Any
  64. def union(first: JavaDoubleRDD, rest: List[JavaDoubleRDD]): JavaDoubleRDD

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes
    JavaSparkContext → JavaSparkContextVarargsWorkaround
  65. def union[K, V](first: JavaPairRDD[K, V], rest: List[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes
    JavaSparkContext → JavaSparkContextVarargsWorkaround
  66. def union[T](first: JavaRDD[T], rest: List[JavaRDD[T]]): JavaRDD[T]

    Build the union of two or more RDDs.

    Build the union of two or more RDDs.

    Definition Classes
    JavaSparkContext → JavaSparkContextVarargsWorkaround
  67. def union[K, V](rdds: <repeated...>[JavaPairRDD[K, V]]): JavaPairRDD[K, V]

    Definition Classes
    JavaSparkContextVarargsWorkaround
  68. def union(rdds: <repeated...>[JavaDoubleRDD]): JavaDoubleRDD

    Definition Classes
    JavaSparkContextVarargsWorkaround
  69. def union[T](rdds: <repeated...>[JavaRDD[T]]): JavaRDD[T]

    Definition Classes
    JavaSparkContextVarargsWorkaround
  70. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  71. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  72. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from JavaSparkContextVarargsWorkaround

Inherited from AnyRef

Inherited from Any

Ungrouped