class InProcessLauncher extends AbstractLauncher[InProcessLauncher]
In-process launcher for Spark applications.
Use this class to start Spark applications programmatically. Applications launched using this class will run in the same process as the caller.
Because Spark only supports a single active instance of SparkContext per JVM, code
that uses this class should be careful about which applications are launched. It's recommended
that this launcher only be used to launch applications in cluster mode.
Also note that, when running applications in client mode, JVM-related configurations (like driver memory or configs which modify the driver's class path) do not take effect. Logging configuration is also inherited from the parent application.
- Source
- InProcessLauncher.java
- Since
2.3.0
- Alphabetic
- By Inheritance
- InProcessLauncher
- AbstractLauncher
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new InProcessLauncher()
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def addAppArgs(args: <repeated...>[String]): InProcessLauncher
Adds command line arguments for the application.
Adds command line arguments for the application.
- args
Arguments to pass to the application's main class.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def addFile(file: String): InProcessLauncher
Adds a file to be submitted with the application.
Adds a file to be submitted with the application.
- file
Path to the file.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def addJar(jar: String): InProcessLauncher
Adds a jar file to be submitted with the application.
Adds a jar file to be submitted with the application.
- jar
Path to the jar file.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def addPyFile(file: String): InProcessLauncher
Adds a python file / zip / egg to be submitted with the application.
Adds a python file / zip / egg to be submitted with the application.
- file
Path to the file.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def addSparkArg(name: String, value: String): InProcessLauncher
Adds an argument with a value to the Spark invocation.
Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.
It is safe to add arguments modified by other methods in this class (such as
#setMaster(String)- the last invocation will be the one to take effect.Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
- name
Name of argument to add.
- value
Value of the argument.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- Since
1.5.0
- def addSparkArg(arg: String): InProcessLauncher
Adds a no-value argument to the Spark invocation.
Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.
Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
- arg
Argument to add.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- Since
1.5.0
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def setAppName(appName: String): InProcessLauncher
Set the application name.
Set the application name.
- appName
Application name.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setAppResource(resource: String): InProcessLauncher
Set the main application resource.
Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.
- resource
Path to the main application resource.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setConf(key: String, value: String): InProcessLauncher
Set a single configuration value for the application.
Set a single configuration value for the application.
- key
Configuration key.
- value
The value to use.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setDeployMode(mode: String): InProcessLauncher
Set the deploy mode for the application.
Set the deploy mode for the application.
- mode
Deploy mode.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setMainClass(mainClass: String): InProcessLauncher
Sets the application class name for Java/Scala applications.
Sets the application class name for Java/Scala applications.
- mainClass
Application's main class.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setMaster(master: String): InProcessLauncher
Set the Spark master for the application.
Set the Spark master for the application.
- master
Spark master.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setPropertiesFile(path: String): InProcessLauncher
Set a custom properties file with Spark configuration for the application.
Set a custom properties file with Spark configuration for the application.
- path
Path to custom properties file to use.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setRemote(remote: String): InProcessLauncher
Set the Spark master for the application.
Set the Spark master for the application.
- remote
Spark remote url.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def setVerbose(verbose: Boolean): InProcessLauncher
Enables verbose reporting for SparkSubmit.
Enables verbose reporting for SparkSubmit.
- verbose
Whether to enable verbose output.
- returns
This launcher.
- Definition Classes
- AbstractLauncher
- def startApplication(listeners: <repeated...>[Listener]): SparkAppHandle
Starts a Spark application.
Starts a Spark application.
- listeners
Listeners to add to the handle before the app is launched.
- returns
A handle for the launched application.
- Definition Classes
- InProcessLauncher → AbstractLauncher
- Annotations
- @Override()
- See also
AbstractLauncher#startApplication(SparkAppHandle.Listener...)
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)