org.apache.spark

Class TestUtils

Object

org.apache.spark.TestUtils

public class TestUtils
extends Object

Utilities for tests. Included in main codebase since it's used by multiple projects.

TODO: See if we can move this to the test codebase by specifying test dependencies between projects.

Constructor Summary

Constructors

Constructor and Description
TestUtils()

Method Summary

Modifier and Type	Method and Description
static <E extends Throwable> void	assertExceptionMsg(Throwable exception, String msg, boolean ignoreCase, scala.reflect.ClassTag<E> evidence$1) Asserts that exception message contains the message.
static void	assertNotSpilled(SparkContext sc, String identifier, scala.Function0<scala.runtime.BoxedUnit> body) Run some code involving jobs submitted to the given context and assert that the jobs did not spill.
static void	assertSpilled(SparkContext sc, String identifier, scala.Function0<scala.runtime.BoxedUnit> body) Run some code involving jobs submitted to the given context and assert that the jobs spilled.
static void	configTestLog4j2(String level) config a log4j2 properties used for testsuite
static java.io.File	createCompiledClass(String className, java.io.File destDir, String toStringValue, String baseClass, scala.collection.Seq<java.net.URL> classpathUrls, scala.collection.Seq<String> implementsClasses, String extraCodeBody) Creates a compiled class with the given name.
static java.io.File	createCompiledClass(String className, java.io.File destDir, org.apache.spark.TestUtils.JavaSourceFromString sourceFile, scala.collection.Seq<java.net.URL> classpathUrls) Creates a compiled class with the source file.
static java.net.URL	createJar(scala.collection.Seq<java.io.File> files, java.io.File jarFile, scala.Option<String> directoryPrefix, scala.Option<String> mainClass) Create a jar file that contains this set of files.
static java.net.URL	createJarWithClasses(scala.collection.Seq<String> classNames, String toStringValue, scala.collection.Seq<scala.Tuple2<String,String>> classNamesWithBase, scala.collection.Seq<java.net.URL> classpathUrls) Create a jar that defines classes with the given names.
static java.net.URL	createJarWithFiles(scala.collection.immutable.Map<String,String> files, java.io.File dir) Create a jar file containing multiple files.
static String	createTempJsonFile(java.io.File dir, String prefix, org.json4s.JsonAST.JValue jsonValue) Creates a temp JSON file that contains the input JSON record.
static String	createTempScriptWithExpectedOutput(java.io.File dir, String prefix, String output) Creates a temp bash script that prints the given output.
static scala.Option<String>	getAbsolutePathFromExecutable(String executable) Get the absolute path from the executable.
static int	httpResponseCode(java.net.URL url, String method, scala.collection.Seq<scala.Tuple2<String,String>> headers) Returns the response code from an HTTP(S) URL.
static String	httpResponseMessage(java.net.URL url, String method, scala.collection.Seq<scala.Tuple2<String,String>> headers) Returns the response message from an HTTP(S) URL.
static boolean	isPythonVersionAvailable()
static String[]	listDirectory(java.io.File path) Returns the list of files at 'path' recursively.
static String	minimumPythonSupportedVersion()
static java.io.File[]	recursiveList(java.io.File f) Lists files recursively.
static boolean	testCommandAvailable(String command) Test if a command is available.
static <T> T	withHttpConnection(java.net.URL url, String method, scala.collection.Seq<scala.Tuple2<String,String>> headers, scala.Function1<java.net.HttpURLConnection,T> fn)
static void	withHttpServer(String resBaseDir, scala.Function1<java.net.URL,scala.runtime.BoxedUnit> body)
static <L extends SparkListener> void	withListener(SparkContext sc, L listener, scala.Function1<L,scala.runtime.BoxedUnit> body) Runs some code with the given listener installed in the SparkContext.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TestUtils

public TestUtils()

Method Detail

createJarWithClasses

public static java.net.URL createJarWithClasses(scala.collection.Seq<String> classNames,
                                                String toStringValue,
                                                scala.collection.Seq<scala.Tuple2<String,String>> classNamesWithBase,
                                                scala.collection.Seq<java.net.URL> classpathUrls)

Create a jar that defines classes with the given names.

Note: if this is used during class loader tests, class names should be unique in order to avoid interference between tests.

Parameters:

classNames - (undocumented)

toStringValue - (undocumented)

classNamesWithBase - (undocumented)

classpathUrls - (undocumented)

Returns:

(undocumented)

createJarWithFiles

public static java.net.URL createJarWithFiles(scala.collection.immutable.Map<String,String> files,
                                              java.io.File dir)

Create a jar file containing multiple files. The files map contains a mapping of file names in the jar file to their contents.

Parameters:

files - (undocumented)

dir - (undocumented)

Returns:

(undocumented)

createJar

public static java.net.URL createJar(scala.collection.Seq<java.io.File> files,
                                     java.io.File jarFile,
                                     scala.Option<String> directoryPrefix,
                                     scala.Option<String> mainClass)

Create a jar file that contains this set of files. All files will be located in the specified directory or at the root of the jar.

Parameters:

files - (undocumented)

jarFile - (undocumented)

directoryPrefix - (undocumented)

mainClass - (undocumented)

Returns:

(undocumented)

createCompiledClass

public static java.io.File createCompiledClass(String className,
                                               java.io.File destDir,
                                               org.apache.spark.TestUtils.JavaSourceFromString sourceFile,
                                               scala.collection.Seq<java.net.URL> classpathUrls)

Creates a compiled class with the source file. Class file will be placed in destDir.

createCompiledClass

public static java.io.File createCompiledClass(String className,
                                               java.io.File destDir,
                                               String toStringValue,
                                               String baseClass,
                                               scala.collection.Seq<java.net.URL> classpathUrls,
                                               scala.collection.Seq<String> implementsClasses,
                                               String extraCodeBody)

Creates a compiled class with the given name. Class file will be placed in destDir.

assertSpilled

public static void assertSpilled(SparkContext sc,
                                 String identifier,
                                 scala.Function0<scala.runtime.BoxedUnit> body)

Run some code involving jobs submitted to the given context and assert that the jobs spilled.

Parameters:

sc - (undocumented)

identifier - (undocumented)

body - (undocumented)

assertNotSpilled

public static void assertNotSpilled(SparkContext sc,
                                    String identifier,
                                    scala.Function0<scala.runtime.BoxedUnit> body)

Run some code involving jobs submitted to the given context and assert that the jobs did not spill.

Parameters:

sc - (undocumented)

identifier - (undocumented)

body - (undocumented)

assertExceptionMsg

public static <E extends Throwable> void assertExceptionMsg(Throwable exception,
                                                            String msg,
                                                            boolean ignoreCase,
                                                            scala.reflect.ClassTag<E> evidence$1)

Asserts that exception message contains the message. Please note this checks all exceptions in the tree. If a type parameter E is supplied, this will additionally confirm that the exception is a subtype of the exception provided in the type parameter.

Parameters:

exception - (undocumented)

msg - (undocumented)

ignoreCase - (undocumented)

evidence$1 - (undocumented)

testCommandAvailable

public static boolean testCommandAvailable(String command)

Test if a command is available.

Parameters:

command - (undocumented)

Returns:

(undocumented)

minimumPythonSupportedVersion

public static String minimumPythonSupportedVersion()

isPythonVersionAvailable

public static boolean isPythonVersionAvailable()

getAbsolutePathFromExecutable

public static scala.Option<String> getAbsolutePathFromExecutable(String executable)

Get the absolute path from the executable. This implementation was borrowed from spark/dev/sparktestsupport/shellutils.py.

Parameters:

executable - (undocumented)

Returns:

(undocumented)

httpResponseCode

public static int httpResponseCode(java.net.URL url,
                                   String method,
                                   scala.collection.Seq<scala.Tuple2<String,String>> headers)

Returns the response code from an HTTP(S) URL.

Parameters:

url - (undocumented)

method - (undocumented)

headers - (undocumented)

Returns:

(undocumented)

httpResponseMessage

public static String httpResponseMessage(java.net.URL url,
                                         String method,
                                         scala.collection.Seq<scala.Tuple2<String,String>> headers)

Returns the response message from an HTTP(S) URL.

Parameters:

url - (undocumented)

method - (undocumented)

headers - (undocumented)

Returns:

(undocumented)

withHttpConnection

public static <T> T withHttpConnection(java.net.URL url,
                                       String method,
                                       scala.collection.Seq<scala.Tuple2<String,String>> headers,
                                       scala.Function1<java.net.HttpURLConnection,T> fn)

withListener

public static <L extends SparkListener> void withListener(SparkContext sc,
                                                          L listener,
                                                          scala.Function1<L,scala.runtime.BoxedUnit> body)

Runs some code with the given listener installed in the SparkContext. After the code runs, this method will wait until all events posted to the listener bus are processed, and then remove the listener from the bus.

Parameters:

sc - (undocumented)

listener - (undocumented)

body - (undocumented)

withHttpServer

public static void withHttpServer(String resBaseDir,
                                  scala.Function1<java.net.URL,scala.runtime.BoxedUnit> body)

configTestLog4j2

public static void configTestLog4j2(String level)

config a log4j2 properties used for testsuite

Parameters:

level - (undocumented)

recursiveList

public static java.io.File[] recursiveList(java.io.File f)

Lists files recursively.

Parameters:

f - (undocumented)

Returns:

(undocumented)

listDirectory

public static String[] listDirectory(java.io.File path)

Returns the list of files at 'path' recursively. This skips files that are ignored normally by MapReduce.

Parameters:

path - (undocumented)

Returns:

(undocumented)

createTempJsonFile

public static String createTempJsonFile(java.io.File dir,
                                        String prefix,
                                        org.json4s.JsonAST.JValue jsonValue)

Creates a temp JSON file that contains the input JSON record.

createTempScriptWithExpectedOutput

public static String createTempScriptWithExpectedOutput(java.io.File dir,
                                                        String prefix,
                                                        String output)

Creates a temp bash script that prints the given output.