Packages

c

org.apache.spark.launcher

SparkLauncher

class SparkLauncher extends AbstractLauncher[SparkLauncher]

Launcher for Spark applications.

Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

Source
SparkLauncher.java
Linear Supertypes
AbstractLauncher[SparkLauncher], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkLauncher
  2. AbstractLauncher
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SparkLauncher(env: Map[String, String])

    Creates a launcher that will set the given environment variables in the child.

    Creates a launcher that will set the given environment variables in the child.

    env

    Environment variables to set.

  2. new SparkLauncher()

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def addAppArgs(args: <repeated...>[String]): SparkLauncher

    Adds command line arguments for the application.

    Adds command line arguments for the application.

    args

    Arguments to pass to the application's main class.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  5. def addFile(file: String): SparkLauncher

    Adds a file to be submitted with the application.

    Adds a file to be submitted with the application.

    file

    Path to the file.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  6. def addJar(jar: String): SparkLauncher

    Adds a jar file to be submitted with the application.

    Adds a jar file to be submitted with the application.

    jar

    Path to the jar file.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  7. def addPyFile(file: String): SparkLauncher

    Adds a python file / zip / egg to be submitted with the application.

    Adds a python file / zip / egg to be submitted with the application.

    file

    Path to the file.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  8. def addSparkArg(name: String, value: String): SparkLauncher

    Adds an argument with a value to the Spark invocation.

    Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.

    It is safe to add arguments modified by other methods in this class (such as #setMaster(String) - the last invocation will be the one to take effect.

    Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

    name

    Name of argument to add.

    value

    Value of the argument.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
    Since

    1.5.0

  9. def addSparkArg(arg: String): SparkLauncher

    Adds a no-value argument to the Spark invocation.

    Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.

    Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.

    arg

    Argument to add.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
    Since

    1.5.0

  10. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  11. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  12. def directory(dir: File): SparkLauncher

    Sets the working directory of spark-submit.

    Sets the working directory of spark-submit.

    dir

    The directory to set as spark-submit's working directory.

    returns

    This launcher.

  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  15. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  16. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def launch(): Process

    Launches a sub-process that will start the configured Spark application.

    Launches a sub-process that will start the configured Spark application.

    The #startApplication(SparkAppHandle.Listener...) method is preferred when launching Spark, since it provides better control of the child application.

    returns

    A process handle for the Spark app.

  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  22. def redirectError(errFile: File): SparkLauncher

    Redirects error output to the specified File.

    Redirects error output to the specified File.

    errFile

    The file to which stderr is written.

    returns

    This launcher.

  23. def redirectError(to: Redirect): SparkLauncher

    Redirects error output to the specified Redirect.

    Redirects error output to the specified Redirect.

    to

    The method of redirection.

    returns

    This launcher.

  24. def redirectError(): SparkLauncher

    Specifies that stderr in spark-submit should be redirected to stdout.

    Specifies that stderr in spark-submit should be redirected to stdout.

    returns

    This launcher.

  25. def redirectOutput(outFile: File): SparkLauncher

    Redirects error output to the specified File.

    Redirects error output to the specified File.

    outFile

    The file to which stdout is written.

    returns

    This launcher.

  26. def redirectOutput(to: Redirect): SparkLauncher

    Redirects standard output to the specified Redirect.

    Redirects standard output to the specified Redirect.

    to

    The method of redirection.

    returns

    This launcher.

  27. def redirectToLog(loggerName: String): SparkLauncher

    Sets all output to be logged and redirected to a logger with the specified name.

    Sets all output to be logged and redirected to a logger with the specified name.

    loggerName

    The name of the logger to log stdout and stderr.

    returns

    This launcher.

  28. def setAppName(appName: String): SparkLauncher

    Set the application name.

    Set the application name.

    appName

    Application name.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  29. def setAppResource(resource: String): SparkLauncher

    Set the main application resource.

    Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.

    resource

    Path to the main application resource.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  30. def setConf(key: String, value: String): SparkLauncher

    Set a single configuration value for the application.

    Set a single configuration value for the application.

    key

    Configuration key.

    value

    The value to use.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  31. def setDeployMode(mode: String): SparkLauncher

    Set the deploy mode for the application.

    Set the deploy mode for the application.

    mode

    Deploy mode.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  32. def setJavaHome(javaHome: String): SparkLauncher

    Set a custom JAVA_HOME for launching the Spark application.

    Set a custom JAVA_HOME for launching the Spark application.

    javaHome

    Path to the JAVA_HOME to use.

    returns

    This launcher.

  33. def setMainClass(mainClass: String): SparkLauncher

    Sets the application class name for Java/Scala applications.

    Sets the application class name for Java/Scala applications.

    mainClass

    Application's main class.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  34. def setMaster(master: String): SparkLauncher

    Set the Spark master for the application.

    Set the Spark master for the application.

    master

    Spark master.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  35. def setPropertiesFile(path: String): SparkLauncher

    Set a custom properties file with Spark configuration for the application.

    Set a custom properties file with Spark configuration for the application.

    path

    Path to custom properties file to use.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  36. def setRemote(remote: String): SparkLauncher

    Set the Spark master for the application.

    Set the Spark master for the application.

    remote

    Spark remote url.

    returns

    This launcher.

    Definition Classes
    AbstractLauncher
  37. def setSparkHome(sparkHome: String): SparkLauncher

    Set a custom Spark installation location for the application.

    Set a custom Spark installation location for the application.

    sparkHome

    Path to the Spark installation to use.

    returns

    This launcher.

  38. def setVerbose(verbose: Boolean): SparkLauncher

    Enables verbose reporting for SparkSubmit.

    Enables verbose reporting for SparkSubmit.

    verbose

    Whether to enable verbose output.

    returns

    This launcher.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
  39. def startApplication(listeners: <repeated...>[Listener]): SparkAppHandle

    Starts a Spark application.

    Starts a Spark application.

    Applications launched by this launcher run as child processes. The child's stdout and stderr are merged and written to a logger (see java.util.logging) only if redirection has not otherwise been configured on this SparkLauncher. The logger's name can be defined by setting #CHILD_PROCESS_LOGGER_NAME in the app's configuration. If that option is not set, the code will try to derive a name from the application's name or main class / script file. If those cannot be determined, an internal, unique name will be used. In all cases, the logger name will start with "org.apache.spark.launcher.app", to fit more easily into the configuration of commonly-used logging systems.

    listeners

    Listeners to add to the handle before the app is launched.

    returns

    A handle for the launched application.

    Definition Classes
    SparkLauncher → AbstractLauncher
    Annotations
    @Override()
    Since

    1.6.0

    See also

    AbstractLauncher#startApplication(SparkAppHandle.Listener...)

  40. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  41. def toString(): String
    Definition Classes
    AnyRef → Any
  42. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  43. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  44. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from AbstractLauncher[SparkLauncher]

Inherited from AnyRef

Inherited from Any

Ungrouped