public class SparkLauncher
extends Object
Modifier and Type | Field and Description |
---|---|
static String |
DRIVER_EXTRA_CLASSPATH
Configuration key for the driver class path.
|
static String |
DRIVER_EXTRA_JAVA_OPTIONS
Configuration key for the driver VM options.
|
static String |
DRIVER_EXTRA_LIBRARY_PATH
Configuration key for the driver native library path.
|
static String |
DRIVER_MEMORY
Configuration key for the driver memory.
|
static String |
EXECUTOR_CORES
Configuration key for the number of executor CPU cores.
|
static String |
EXECUTOR_EXTRA_CLASSPATH
Configuration key for the executor class path.
|
static String |
EXECUTOR_EXTRA_JAVA_OPTIONS
Configuration key for the executor VM options.
|
static String |
EXECUTOR_EXTRA_LIBRARY_PATH
Configuration key for the executor native library path.
|
static String |
EXECUTOR_MEMORY
Configuration key for the executor memory.
|
static String |
SPARK_MASTER
The Spark master.
|
Constructor and Description |
---|
SparkLauncher() |
SparkLauncher(java.util.Map<String,String> env)
Creates a launcher that will set the given environment variables in the child.
|
Modifier and Type | Method and Description |
---|---|
SparkLauncher |
addAppArgs(String... args)
Adds command line arguments for the application.
|
SparkLauncher |
addFile(String file)
Adds a file to be submitted with the application.
|
SparkLauncher |
addJar(String jar)
Adds a jar file to be submitted with the application.
|
SparkLauncher |
addPyFile(String file)
Adds a python file / zip / egg to be submitted with the application.
|
Process |
launch()
Launches a sub-process that will start the configured Spark application.
|
SparkLauncher |
setAppName(String appName)
Set the application name.
|
SparkLauncher |
setAppResource(String resource)
Set the main application resource.
|
SparkLauncher |
setConf(String key,
String value)
Set a single configuration value for the application.
|
SparkLauncher |
setDeployMode(String mode)
Set the deploy mode for the application.
|
SparkLauncher |
setJavaHome(String javaHome)
Set a custom JAVA_HOME for launching the Spark application.
|
SparkLauncher |
setMainClass(String mainClass)
Sets the application class name for Java/Scala applications.
|
SparkLauncher |
setMaster(String master)
Set the Spark master for the application.
|
SparkLauncher |
setPropertiesFile(String path)
Set a custom properties file with Spark configuration for the application.
|
SparkLauncher |
setSparkHome(String sparkHome)
Set a custom Spark installation location for the application.
|
SparkLauncher |
setVerbose(boolean verbose)
Enables verbose reporting for SparkSubmit.
|
public static final String SPARK_MASTER
public static final String DRIVER_MEMORY
public static final String DRIVER_EXTRA_CLASSPATH
public static final String DRIVER_EXTRA_JAVA_OPTIONS
public static final String DRIVER_EXTRA_LIBRARY_PATH
public static final String EXECUTOR_MEMORY
public static final String EXECUTOR_EXTRA_CLASSPATH
public static final String EXECUTOR_EXTRA_JAVA_OPTIONS
public static final String EXECUTOR_EXTRA_LIBRARY_PATH
public static final String EXECUTOR_CORES
public SparkLauncher()
public SparkLauncher(java.util.Map<String,String> env)
env
- Environment variables to set.public SparkLauncher setJavaHome(String javaHome)
javaHome
- Path to the JAVA_HOME to use.public SparkLauncher setSparkHome(String sparkHome)
sparkHome
- Path to the Spark installation to use.public SparkLauncher setPropertiesFile(String path)
path
- Path to custom properties file to use.public SparkLauncher setConf(String key, String value)
key
- Configuration key.value
- The value to use.public SparkLauncher setAppName(String appName)
appName
- Application name.public SparkLauncher setMaster(String master)
master
- Spark master.public SparkLauncher setDeployMode(String mode)
mode
- Deploy mode.public SparkLauncher setAppResource(String resource)
resource
- Path to the main application resource.public SparkLauncher setMainClass(String mainClass)
mainClass
- Application's main class.public SparkLauncher addAppArgs(String... args)
args
- Arguments to pass to the application's main class.public SparkLauncher addJar(String jar)
jar
- Path to the jar file.public SparkLauncher addFile(String file)
file
- Path to the file.public SparkLauncher addPyFile(String file)
file
- Path to the file.public SparkLauncher setVerbose(boolean verbose)
verbose
- Whether to enable verbose output.public Process launch() throws java.io.IOException
java.io.IOException