org.apache.spark.scheduler

JobLogger

class JobLogger extends SparkListener with Logging

:: DeveloperApi :: A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext

NOTE: The functionality of this class is heavily stripped down to accommodate for a general refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced to log application information as SparkListenerEvents. To enable this functionality, set spark.eventLog.enabled to true.

Annotations
@DeveloperApi() @deprecated
Deprecated

(Since version 1.0.0) Log application information by setting spark.eventLog.enabled.

Source
JobLogger.scala
Linear Supertypes
Logging, SparkListener, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. JobLogger
  2. Logging
  3. SparkListener
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JobLogger()

  2. new JobLogger(user: String, logDirName: String)

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def buildJobStageDependencies(jobId: Int, stageIds: Seq[Int]): Unit

    Build up the maps that represent stage-job relationships

    Build up the maps that represent stage-job relationships

    jobId

    ID of the job

    stageIds

    IDs of the associated stages

    Attributes
    protected
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def closeLogWriter(jobId: Int): Unit

    Close log file, and clean the stage relationship in stageIdToJobId

    Close log file, and clean the stage relationship in stageIdToJobId

    jobId

    ID of the job

    Attributes
    protected
  10. def createLogDir(): Unit

    Create a folder for log files, the folder's name is the creation time of jobLogger

    Create a folder for log files, the folder's name is the creation time of jobLogger

    Attributes
    protected
  11. def createLogWriter(jobId: Int): Unit

    Create a log file for one job

    Create a log file for one job

    jobId

    ID of the job

    Attributes
    protected
    Exceptions thrown
    FileNotFoundException

    Fail to create log file

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  16. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  17. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  18. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  19. def jobLogInfo(jobId: Int, info: String, withTime: Boolean = true): Unit

    Write info into log file

    Write info into log file

    jobId

    ID of the job

    info

    Info to be recorded

    withTime

    Controls whether to record time stamp before the info, default is true

    Attributes
    protected
  20. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  21. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  22. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  23. val logDirName: String

  24. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  25. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  26. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  28. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  29. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  31. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  32. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  33. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  34. final def notify(): Unit

    Definition Classes
    AnyRef
  35. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  36. def onApplicationEnd(applicationEnd: SparkListenerApplicationEnd): Unit

    Called when the application ends

    Called when the application ends

    Definition Classes
    SparkListener
  37. def onApplicationStart(applicationStart: SparkListenerApplicationStart): Unit

    Called when the application starts

    Called when the application starts

    Definition Classes
    SparkListener
  38. def onBlockManagerAdded(blockManagerAdded: SparkListenerBlockManagerAdded): Unit

    Called when a new block manager has joined

    Called when a new block manager has joined

    Definition Classes
    SparkListener
  39. def onBlockManagerRemoved(blockManagerRemoved: SparkListenerBlockManagerRemoved): Unit

    Called when an existing block manager has been removed

    Called when an existing block manager has been removed

    Definition Classes
    SparkListener
  40. def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated): Unit

    Called when the driver receives a block update info.

    Called when the driver receives a block update info.

    Definition Classes
    SparkListener
  41. def onEnvironmentUpdate(environmentUpdate: SparkListenerEnvironmentUpdate): Unit

    Called when environment properties have been updated

    Called when environment properties have been updated

    Definition Classes
    SparkListener
  42. def onExecutorAdded(executorAdded: SparkListenerExecutorAdded): Unit

    Called when the driver registers a new executor.

    Called when the driver registers a new executor.

    Definition Classes
    SparkListener
  43. def onExecutorMetricsUpdate(executorMetricsUpdate: SparkListenerExecutorMetricsUpdate): Unit

    Called when the driver receives task metrics from an executor in a heartbeat.

    Called when the driver receives task metrics from an executor in a heartbeat.

    Definition Classes
    SparkListener
  44. def onExecutorRemoved(executorRemoved: SparkListenerExecutorRemoved): Unit

    Called when the driver removes an executor.

    Called when the driver removes an executor.

    Definition Classes
    SparkListener
  45. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit

    When job ends, recording job completion status and close log file

    When job ends, recording job completion status and close log file

    jobEnd

    Job end event

    Definition Classes
    JobLoggerSparkListener
  46. def onJobStart(jobStart: SparkListenerJobStart): Unit

    When job starts, record job property and stage graph

    When job starts, record job property and stage graph

    jobStart

    Job start event

    Definition Classes
    JobLoggerSparkListener
  47. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit

    When stage is completed, record stage completion status

    When stage is completed, record stage completion status

    stageCompleted

    Stage completed event

    Definition Classes
    JobLoggerSparkListener
  48. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit

    When stage is submitted, record stage submit info

    When stage is submitted, record stage submit info

    stageSubmitted

    Stage submitted event

    Definition Classes
    JobLoggerSparkListener
  49. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit

    When task ends, record task completion status and metrics

    When task ends, record task completion status and metrics

    taskEnd

    Task end event

    Definition Classes
    JobLoggerSparkListener
  50. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit

    Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

    Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

    Definition Classes
    SparkListener
  51. def onTaskStart(taskStart: SparkListenerTaskStart): Unit

    Called when a task starts

    Called when a task starts

    Definition Classes
    SparkListener
  52. def onUnpersistRDD(unpersistRDD: SparkListenerUnpersistRDD): Unit

    Called when an RDD is manually unpersisted by the application

    Called when an RDD is manually unpersisted by the application

    Definition Classes
    SparkListener
  53. def recordJobProperties(jobId: Int, properties: Properties): Unit

    Record job properties into job log file

    Record job properties into job log file

    jobId

    ID of the job

    properties

    Properties of the job

    Attributes
    protected
  54. def recordTaskMetrics(stageId: Int, status: String, taskInfo: TaskInfo, taskMetrics: TaskMetrics): Unit

    Record task metrics into job log files, including execution info and shuffle metrics

    Record task metrics into job log files, including execution info and shuffle metrics

    stageId

    Stage ID of the task

    status

    Status info of the task

    taskInfo

    Task description info

    taskMetrics

    Task running metrics

    Attributes
    protected
  55. def stageLogInfo(stageId: Int, info: String, withTime: Boolean = true): Unit

    Write info into log file

    Write info into log file

    stageId

    ID of the stage

    info

    Info to be recorded

    withTime

    Controls whether to record time stamp before the info, default is true

    Attributes
    protected
  56. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  57. def toString(): String

    Definition Classes
    AnyRef → Any
  58. val user: String

  59. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  60. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  61. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from SparkListener

Inherited from AnyRef

Inherited from Any

Ungrouped