public class EventLoggingListener extends Object implements SparkListener, Logging
Event logging is specified by the following configurable parameters: spark.eventLog.enabled - Whether event logging is enabled. spark.eventLog.compress - Whether to compress logged events spark.eventLog.overwrite - Whether to overwrite any existing files. spark.eventLog.dir - Path to the directory in which events are logged. spark.eventLog.buffer.kb - Buffer size to use when writing to output streams
Constructor and Description |
---|
EventLoggingListener(String appId,
java.net.URI logBaseDir,
SparkConf sparkConf) |
EventLoggingListener(String appId,
java.net.URI logBaseDir,
SparkConf sparkConf,
org.apache.hadoop.conf.Configuration hadoopConf) |
Modifier and Type | Method and Description |
---|---|
static String |
COMPRESSION_CODEC_KEY() |
static String |
DEFAULT_LOG_DIR() |
static String |
getLogPath(java.net.URI logBaseDir,
String appId,
scala.Option<String> compressionCodecName)
Return a file-system-safe path to the log file for the given application.
|
static String |
IN_PROGRESS() |
static void |
initEventLog(java.io.OutputStream logStream)
Write metadata about an event log to the given stream.
|
scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue> |
loggedEvents() |
String |
logPath() |
void |
onApplicationEnd(SparkListenerApplicationEnd event)
Called when the application ends
|
void |
onApplicationStart(SparkListenerApplicationStart event)
Called when the application starts
|
void |
onBlockManagerAdded(SparkListenerBlockManagerAdded event)
Called when a new block manager has joined
|
void |
onBlockManagerRemoved(SparkListenerBlockManagerRemoved event)
Called when an existing block manager has been removed
|
void |
onEnvironmentUpdate(SparkListenerEnvironmentUpdate event)
Called when environment properties have been updated
|
void |
onExecutorAdded(SparkListenerExecutorAdded event)
Called when the driver registers a new executor.
|
void |
onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event)
Called when the driver receives task metrics from an executor in a heartbeat.
|
void |
onExecutorRemoved(SparkListenerExecutorRemoved event)
Called when the driver removes an executor.
|
void |
onJobEnd(SparkListenerJobEnd event)
Called when a job ends
|
void |
onJobStart(SparkListenerJobStart event)
Called when a job starts
|
void |
onStageCompleted(SparkListenerStageCompleted event)
Called when a stage completes successfully or fails, with information on the completed stage.
|
void |
onStageSubmitted(SparkListenerStageSubmitted event)
Called when a stage is submitted
|
void |
onTaskEnd(SparkListenerTaskEnd event)
Called when a task ends
|
void |
onTaskGettingResult(SparkListenerTaskGettingResult event)
Called when a task begins remotely fetching its result (will not be called for tasks that do
not need to fetch the result remotely).
|
void |
onTaskStart(SparkListenerTaskStart event)
Called when a task starts
|
void |
onUnpersistRDD(SparkListenerUnpersistRDD event)
Called when an RDD is manually unpersisted by the application
|
static java.io.InputStream |
openEventLog(org.apache.hadoop.fs.Path log,
org.apache.hadoop.fs.FileSystem fs)
Opens an event log file and returns an input stream that contains the event data.
|
static String |
SPARK_VERSION_KEY() |
void |
start()
Creates the log file in the configured log directory.
|
boolean |
stop()
Stop logging events.
|
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
public EventLoggingListener(String appId, java.net.URI logBaseDir, SparkConf sparkConf, org.apache.hadoop.conf.Configuration hadoopConf)
public EventLoggingListener(String appId, java.net.URI logBaseDir, SparkConf sparkConf)
public static String IN_PROGRESS()
public static String DEFAULT_LOG_DIR()
public static String SPARK_VERSION_KEY()
public static String COMPRESSION_CODEC_KEY()
public static void initEventLog(java.io.OutputStream logStream)
logStream
- Raw output stream to the event log file.public static String getLogPath(java.net.URI logBaseDir, String appId, scala.Option<String> compressionCodecName)
Note that because we currently only create a single log file for each application, we must encode all the information needed to parse this event log in the file name instead of within the file itself. Otherwise, if the file is compressed, for instance, we won't know which codec to use to decompress the metadata needed to open the file in the first place.
logBaseDir
- Directory where the log file will be written.appId
- A unique app ID.compressionCodecName
- Name to identify the codec used to compress the contents
of the log, or None if compression is not enabled.public static java.io.InputStream openEventLog(org.apache.hadoop.fs.Path log, org.apache.hadoop.fs.FileSystem fs)
public scala.collection.mutable.ArrayBuffer<org.json4s.JsonAST.JValue> loggedEvents()
public String logPath()
public void start()
public void onStageSubmitted(SparkListenerStageSubmitted event)
SparkListener
onStageSubmitted
in interface SparkListener
public void onTaskStart(SparkListenerTaskStart event)
SparkListener
onTaskStart
in interface SparkListener
public void onTaskGettingResult(SparkListenerTaskGettingResult event)
SparkListener
onTaskGettingResult
in interface SparkListener
public void onTaskEnd(SparkListenerTaskEnd event)
SparkListener
onTaskEnd
in interface SparkListener
public void onEnvironmentUpdate(SparkListenerEnvironmentUpdate event)
SparkListener
onEnvironmentUpdate
in interface SparkListener
public void onStageCompleted(SparkListenerStageCompleted event)
SparkListener
onStageCompleted
in interface SparkListener
public void onJobStart(SparkListenerJobStart event)
SparkListener
onJobStart
in interface SparkListener
public void onJobEnd(SparkListenerJobEnd event)
SparkListener
onJobEnd
in interface SparkListener
public void onBlockManagerAdded(SparkListenerBlockManagerAdded event)
SparkListener
onBlockManagerAdded
in interface SparkListener
public void onBlockManagerRemoved(SparkListenerBlockManagerRemoved event)
SparkListener
onBlockManagerRemoved
in interface SparkListener
public void onUnpersistRDD(SparkListenerUnpersistRDD event)
SparkListener
onUnpersistRDD
in interface SparkListener
public void onApplicationStart(SparkListenerApplicationStart event)
SparkListener
onApplicationStart
in interface SparkListener
public void onApplicationEnd(SparkListenerApplicationEnd event)
SparkListener
onApplicationEnd
in interface SparkListener
public void onExecutorAdded(SparkListenerExecutorAdded event)
SparkListener
onExecutorAdded
in interface SparkListener
public void onExecutorRemoved(SparkListenerExecutorRemoved event)
SparkListener
onExecutorRemoved
in interface SparkListener
public void onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate event)
SparkListener
onExecutorMetricsUpdate
in interface SparkListener
public boolean stop()