org.apache.spark.scheduler

JobLogger

class JobLogger extends SparkListener with Logging

A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing RDD graph, tasks start/stop, shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext

Linear Supertypes
Logging, SparkListener, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. JobLogger
  2. Logging
  3. SparkListener
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new JobLogger()

  2. new JobLogger(user: String, logDirName: String)

    logDirName

    The base directory for the log files.

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def buildJobDep(jobID: Int, stage: Stage): Unit

    Build stage dependency for a job

    Build stage dependency for a job

    jobID

    ID of the job

    stage

    Root stage of the job

    Attributes
    protected
  8. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  9. def closeLogWriter(jobID: Int): Unit

    Close log file, and clean the stage relationship in stageIDToJobID

    Close log file, and clean the stage relationship in stageIDToJobID

    jobID

    ID of the job

    Attributes
    protected
  10. def createLogDir(): Unit

    Create a folder for log files, the folder's name is the creation time of jobLogger

    Create a folder for log files, the folder's name is the creation time of jobLogger

    Attributes
    protected
  11. def createLogWriter(jobID: Int): Unit

    Create a log file for one job

    Create a log file for one job

    jobID

    ID of the job

    Attributes
    protected
  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  13. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  16. def getRddName(rdd: RDD[_]): String

    Get RDD's name

    Get RDD's name

    rdd

    Input RDD

    returns

    String of RDD's name

    Attributes
    protected
  17. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  18. def indentString(indent: Int): String

    Generate indents and convert to String

    Generate indents and convert to String

    indent

    Number of indents

    returns

    string of indents

    Attributes
    protected
  19. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  20. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  21. def jobLogInfo(jobID: Int, info: String, withTime: Boolean = true): Unit

    Write info into log file

    Write info into log file

    jobID

    ID of the job

    info

    Info to be recorded

    withTime

    Controls whether to record time stamp before the info, default is true

    Attributes
    protected
  22. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  23. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  24. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  25. val logDirName: String

    The base directory for the log files.

  26. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  27. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  29. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  31. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  32. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  33. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  34. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  37. def onJobEnd(jobEnd: SparkListenerJobEnd): Unit

    When job ends, recording job completion status and close log file

    When job ends, recording job completion status and close log file

    jobEnd

    Job end event

    Definition Classes
    JobLoggerSparkListener
  38. def onJobStart(jobStart: SparkListenerJobStart): Unit

    When job starts, record job property and stage graph

    When job starts, record job property and stage graph

    jobStart

    Job start event

    Definition Classes
    JobLoggerSparkListener
  39. def onStageCompleted(stageCompleted: SparkListenerStageCompleted): Unit

    When stage is completed, record stage completion status

    When stage is completed, record stage completion status

    stageCompleted

    Stage completed event

    Definition Classes
    JobLoggerSparkListener
  40. def onStageSubmitted(stageSubmitted: SparkListenerStageSubmitted): Unit

    When stage is submitted, record stage submit info

    When stage is submitted, record stage submit info

    stageSubmitted

    Stage submitted event

    Definition Classes
    JobLoggerSparkListener
  41. def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit

    When task ends, record task completion status and metrics

    When task ends, record task completion status and metrics

    taskEnd

    Task end event

    Definition Classes
    JobLoggerSparkListener
  42. def onTaskGettingResult(taskGettingResult: SparkListenerTaskGettingResult): Unit

    Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

    Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).

    Definition Classes
    SparkListener
  43. def onTaskStart(taskStart: SparkListenerTaskStart): Unit

    Called when a task starts

    Called when a task starts

    Definition Classes
    JobLoggerSparkListener
  44. def recordJobProperties(jobID: Int, properties: Properties): Unit

    Record job properties into job log file

    Record job properties into job log file

    jobID

    ID of the job

    properties

    Properties of the job

    Attributes
    protected
  45. def recordRddInStageGraph(jobID: Int, rdd: RDD[_], indent: Int): Unit

    Record RDD dependency graph in a stage

    Record RDD dependency graph in a stage

    jobID

    Job ID of the stage

    rdd

    Root RDD of the stage

    indent

    Indent number before info

    Attributes
    protected
  46. def recordStageDep(jobID: Int): Unit

    Record stage dependency and RDD dependency for a stage

    Record stage dependency and RDD dependency for a stage

    jobID

    Job ID of the stage

    Attributes
    protected
  47. def recordStageDepGraph(jobID: Int, stage: Stage, idSet: HashSet[Int], indent: Int = 0): Unit

    Record stage dependency graph of a job

    Record stage dependency graph of a job

    jobID

    Job ID of the stage

    stage

    Root stage of the job

    indent

    Indent number before info, default is 0

    Attributes
    protected
  48. def recordTaskMetrics(stageID: Int, status: String, taskInfo: TaskInfo, taskMetrics: TaskMetrics): Unit

    Record task metrics into job log files, including execution info and shuffle metrics

    Record task metrics into job log files, including execution info and shuffle metrics

    stageID

    Stage ID of the task

    status

    Status info of the task

    taskInfo

    Task description info

    taskMetrics

    Task running metrics

    Attributes
    protected
  49. def stageLogInfo(stageID: Int, info: String, withTime: Boolean = true): Unit

    Write info into log file

    Write info into log file

    stageID

    ID of the stage

    info

    Info to be recorded

    withTime

    Controls whether to record time stamp before the info, default is true

    Attributes
    protected
  50. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  51. def toString(): String

    Definition Classes
    AnyRef → Any
  52. val user: String

  53. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  54. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  55. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Logging

Inherited from SparkListener

Inherited from AnyRef

Inherited from Any

Ungrouped