org.apache.spark.launcher
Class SparkLauncher

Object
  org.apache.spark.launcher.SparkLauncher

public class SparkLauncher
extends Object

Launcher for Spark applications.

Use this class to start Spark applications programmatically. The class uses a builder pattern to allow clients to configure the Spark application and launch it as a child process.

Field Summary

static String	DRIVER_EXTRA_CLASSPATH Configuration key for the driver class path.
static String	DRIVER_EXTRA_JAVA_OPTIONS Configuration key for the driver VM options.
static String	DRIVER_EXTRA_LIBRARY_PATH Configuration key for the driver native library path.
static String	DRIVER_MEMORY Configuration key for the driver memory.
static String	EXECUTOR_CORES Configuration key for the number of executor CPU cores.
static String	EXECUTOR_EXTRA_CLASSPATH Configuration key for the executor class path.
static String	EXECUTOR_EXTRA_JAVA_OPTIONS Configuration key for the executor VM options.
static String	EXECUTOR_EXTRA_LIBRARY_PATH Configuration key for the executor native library path.
static String	EXECUTOR_MEMORY Configuration key for the executor memory.
static String	SPARK_MASTER The Spark master.

Constructor Summary

SparkLauncher()

SparkLauncher(java.util.Map<String,String> env)
Creates a launcher that will set the given environment variables in the child.

Method Summary

SparkLauncher	addAppArgs(String... args) Adds command line arguments for the application.
SparkLauncher	addFile(String file) Adds a file to be submitted with the application.
SparkLauncher	addJar(String jar) Adds a jar file to be submitted with the application.
SparkLauncher	addPyFile(String file) Adds a python file / zip / egg to be submitted with the application.
Process	launch() Launches a sub-process that will start the configured Spark application.
SparkLauncher	setAppName(String appName) Set the application name.
SparkLauncher	setAppResource(String resource) Set the main application resource.
SparkLauncher	setConf(String key, String value) Set a single configuration value for the application.
SparkLauncher	setDeployMode(String mode) Set the deploy mode for the application.
SparkLauncher	setJavaHome(String javaHome) Set a custom JAVA_HOME for launching the Spark application.
SparkLauncher	setMainClass(String mainClass) Sets the application class name for Java/Scala applications.
SparkLauncher	setMaster(String master) Set the Spark master for the application.
SparkLauncher	setPropertiesFile(String path) Set a custom properties file with Spark configuration for the application.
SparkLauncher	setSparkHome(String sparkHome) Set a custom Spark installation location for the application.
SparkLauncher	setVerbose(boolean verbose) Enables verbose reporting for SparkSubmit.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPARK_MASTER

public static final String SPARK_MASTER

The Spark master.

See Also:

Constant Field Values

DRIVER_MEMORY

public static final String DRIVER_MEMORY

Configuration key for the driver memory.

See Also:

Constant Field Values

DRIVER_EXTRA_CLASSPATH

public static final String DRIVER_EXTRA_CLASSPATH

Configuration key for the driver class path.

See Also:

Constant Field Values

DRIVER_EXTRA_JAVA_OPTIONS

public static final String DRIVER_EXTRA_JAVA_OPTIONS

Configuration key for the driver VM options.

See Also:

Constant Field Values

DRIVER_EXTRA_LIBRARY_PATH

public static final String DRIVER_EXTRA_LIBRARY_PATH

Configuration key for the driver native library path.

See Also:

Constant Field Values

EXECUTOR_MEMORY

public static final String EXECUTOR_MEMORY

Configuration key for the executor memory.

See Also:

Constant Field Values

EXECUTOR_EXTRA_CLASSPATH

public static final String EXECUTOR_EXTRA_CLASSPATH

Configuration key for the executor class path.

See Also:

Constant Field Values

EXECUTOR_EXTRA_JAVA_OPTIONS

public static final String EXECUTOR_EXTRA_JAVA_OPTIONS

Configuration key for the executor VM options.

See Also:

Constant Field Values

EXECUTOR_EXTRA_LIBRARY_PATH

public static final String EXECUTOR_EXTRA_LIBRARY_PATH

Configuration key for the executor native library path.

See Also:

Constant Field Values

EXECUTOR_CORES

public static final String EXECUTOR_CORES

Configuration key for the number of executor CPU cores.

See Also:

Constant Field Values

Constructor Detail

SparkLauncher

public SparkLauncher()

SparkLauncher

public SparkLauncher(java.util.Map<String,String> env)

Creates a launcher that will set the given environment variables in the child.

Parameters:

env - Environment variables to set.

Method Detail

setJavaHome

public SparkLauncher setJavaHome(String javaHome)

Set a custom JAVA_HOME for launching the Spark application.

Parameters:

javaHome - Path to the JAVA_HOME to use.

Returns:

This launcher.

setSparkHome

public SparkLauncher setSparkHome(String sparkHome)

Set a custom Spark installation location for the application.

Parameters:

sparkHome - Path to the Spark installation to use.

Returns:

This launcher.

setPropertiesFile

public SparkLauncher setPropertiesFile(String path)

Set a custom properties file with Spark configuration for the application.

Parameters:

path - Path to custom properties file to use.

Returns:

This launcher.

setConf

public SparkLauncher setConf(String key,
                             String value)

Set a single configuration value for the application.

Parameters:

key - Configuration key.

value - The value to use.

Returns:

This launcher.

setAppName

public SparkLauncher setAppName(String appName)

Set the application name.

Parameters:

appName - Application name.

Returns:

This launcher.

setMaster

public SparkLauncher setMaster(String master)

Set the Spark master for the application.

Parameters:

master - Spark master.

Returns:

This launcher.

setDeployMode

public SparkLauncher setDeployMode(String mode)

Set the deploy mode for the application.

Parameters:

mode - Deploy mode.

Returns:

This launcher.

setAppResource

public SparkLauncher setAppResource(String resource)

Set the main application resource. This should be the location of a jar file for Scala/Java applications, or a python script for PySpark applications.

Parameters:

resource - Path to the main application resource.

Returns:

This launcher.

setMainClass

public SparkLauncher setMainClass(String mainClass)

Sets the application class name for Java/Scala applications.

Parameters:

mainClass - Application's main class.

Returns:

This launcher.

addAppArgs

public SparkLauncher addAppArgs(String... args)

Adds command line arguments for the application.

Parameters:

args - Arguments to pass to the application's main class.

Returns:

This launcher.

addJar

public SparkLauncher addJar(String jar)

Adds a jar file to be submitted with the application.

Parameters:

jar - Path to the jar file.

Returns:

This launcher.

addFile

public SparkLauncher addFile(String file)

Adds a file to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

addPyFile

public SparkLauncher addPyFile(String file)

Adds a python file / zip / egg to be submitted with the application.

Parameters:

file - Path to the file.

Returns:

This launcher.

setVerbose

public SparkLauncher setVerbose(boolean verbose)

Enables verbose reporting for SparkSubmit.

Parameters:

verbose - Whether to enable verbose output.

Returns:

This launcher.

launch

public Process launch()
               throws java.io.IOException

Launches a sub-process that will start the configured Spark application.

Returns:

A process handle for the Spark app.

Throws:

java.io.IOException