abstract class AbstractLauncher[T <: AbstractLauncher[T]] extends AnyRef
Base class for launcher implementations.
- Source
- AbstractLauncher.java
- Since
2.3.0
- Alphabetic
- By Inheritance
- AbstractLauncher
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Abstract Value Members
- abstract def startApplication(listeners: <repeated...>[Listener]): SparkAppHandle
Starts a Spark application.
Starts a Spark application.
This method returns a handle that provides information about the running application and can be used to do basic interaction with it.
The returned handle assumes that the application will instantiate a single SparkContext during its lifetime. Once that context reports a final state (one that indicates the SparkContext has stopped), the handle will not perform new state transitions, so anything that happens after that cannot be monitored. If the underlying application is launched as a child process,
SparkAppHandle#kill()can still be used to kill the child process.- listeners
Listeners to add to the handle before the app is launched.
- returns
A handle for the launched application.
- Since
1.6.0
Concrete Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def addAppArgs(args: <repeated...>[String]): T
Adds command line arguments for the application.
Adds command line arguments for the application.
- args
Arguments to pass to the application's main class.
- returns
This launcher.
- def addFile(file: String): T
Adds a file to be submitted with the application.
Adds a file to be submitted with the application.
- file
Path to the file.
- returns
This launcher.
- def addJar(jar: String): T
Adds a jar file to be submitted with the application.
Adds a jar file to be submitted with the application.
- jar
Path to the jar file.
- returns
This launcher.
- def addPyFile(file: String): T
Adds a python file / zip / egg to be submitted with the application.
Adds a python file / zip / egg to be submitted with the application.
- file
Path to the file.
- returns
This launcher.
- def addSparkArg(name: String, value: String): T
Adds an argument with a value to the Spark invocation.
Adds an argument with a value to the Spark invocation. If the argument name corresponds to a known argument, the code validates that the argument actually expects a value, and throws an exception otherwise.
It is safe to add arguments modified by other methods in this class (such as
#setMaster(String)- the last invocation will be the one to take effect.Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
- name
Name of argument to add.
- value
Value of the argument.
- returns
This launcher.
- Since
1.5.0
- def addSparkArg(arg: String): T
Adds a no-value argument to the Spark invocation.
Adds a no-value argument to the Spark invocation. If the argument is known, this method validates whether the argument is indeed a no-value argument, and throws an exception otherwise.
Use this method with caution. It is possible to create an invalid Spark command by passing unknown arguments to this method, since those are allowed for forward compatibility.
- arg
Argument to add.
- returns
This launcher.
- Since
1.5.0
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @IntrinsicCandidate() @native()
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @IntrinsicCandidate() @native()
- def setAppName(appName: String): T
Set the application name.
Set the application name.
- appName
Application name.
- returns
This launcher.
- def setAppResource(resource: String): T
Set the main application resource.
Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.
- resource
Path to the main application resource.
- returns
This launcher.
- def setConf(key: String, value: String): T
Set a single configuration value for the application.
Set a single configuration value for the application.
- key
Configuration key.
- value
The value to use.
- returns
This launcher.
- def setDeployMode(mode: String): T
Set the deploy mode for the application.
Set the deploy mode for the application.
- mode
Deploy mode.
- returns
This launcher.
- def setMainClass(mainClass: String): T
Sets the application class name for Java/Scala applications.
Sets the application class name for Java/Scala applications.
- mainClass
Application's main class.
- returns
This launcher.
- def setMaster(master: String): T
Set the Spark master for the application.
Set the Spark master for the application.
- master
Spark master.
- returns
This launcher.
- def setPropertiesFile(path: String): T
Set a custom properties file with Spark configuration for the application.
Set a custom properties file with Spark configuration for the application.
- path
Path to custom properties file to use.
- returns
This launcher.
- def setRemote(remote: String): T
Set the Spark master for the application.
Set the Spark master for the application.
- remote
Spark remote url.
- returns
This launcher.
- def setVerbose(verbose: Boolean): T
Enables verbose reporting for SparkSubmit.
Enables verbose reporting for SparkSubmit.
- verbose
Whether to enable verbose output.
- returns
This launcher.
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)