Packages

c

org.apache.spark.sql.pipelines.logging

FlowProgressEventLogger

class FlowProgressEventLogger extends Logging

This class should be used for all flow progress events logging, it controls the level at which events are logged. It uses execution mode, flow name and previous flow statuses to infer the level at which an event is to be logged. Below is a more details description of how flow progress events for batch/streaming flows will be logged:

For batch & streaming flows in triggered execution mode:

  • All flow progress events other than errors/warnings will be logged at INFO level (including flow progress events with metrics) and error/warning messages will be logged at their level.
Source
FlowProgressEventLogger.scala
Linear Supertypes
Logging, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. FlowProgressEventLogger
  2. Logging
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new FlowProgressEventLogger(eventCallback: (PipelineEvent) => Unit)

    eventCallback

    Callback to invoke on the flow progress events.

Type Members

  1. implicit class LogStringContext extends AnyRef
    Definition Classes
    Logging

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def MDC(key: LogKey, value: Any): MDC
    Attributes
    protected
    Definition Classes
    Logging
  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @IntrinsicCandidate() @native()
  11. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  12. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  15. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  16. def logBasedOnLevel(level: Level)(f: => MessageWithContext): Unit
    Attributes
    protected
    Definition Classes
    Logging
  17. def logDebug(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  18. def logDebug(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  19. def logDebug(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  20. def logDebug(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  21. def logError(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  22. def logError(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  23. def logError(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  24. def logError(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  25. def logInfo(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  26. def logInfo(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  27. def logInfo(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  28. def logInfo(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  29. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  30. def logTrace(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  31. def logTrace(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  32. def logTrace(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  33. def logTrace(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  34. def logWarning(msg: => String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  35. def logWarning(entry: LogEntry, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  36. def logWarning(entry: LogEntry): Unit
    Attributes
    protected
    Definition Classes
    Logging
  37. def logWarning(msg: => String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  38. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  39. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  40. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @IntrinsicCandidate() @native()
  41. def recordCompletion(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as COMPLETED.

    Records flow progress events with flow status as COMPLETED. For batch flows in continuous mode, events will be logged at METRICS since a completed status is always preceded by running status.

    Note that flow complete events for batch flows are expected to contain quality stats where as for streaming flows quality stats are not expected and hence not added to the flow progress event.

  42. def recordExcluded(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as EXCLUDED at INFO level.

  43. def recordFailed(flow: ResolutionCompletedFlow, exception: Throwable, logAsWarn: Boolean, messageOpt: Option[String] = None): Unit

    Records flow progress events with failure flow status.

    Records flow progress events with failure flow status. By default failed flow progress events are logged at ERROR level, logAsWarn serve as a way to log the event as a WARN.

  44. def recordIdle(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as IDLE.

  45. def recordPlanningForBatchFlow(batchFlow: ResolvedFlow): Unit

    Records flow progress events with flow status as PLANNING for batch flows.

  46. def recordQueued(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as QUEUED.

    Records flow progress events with flow status as QUEUED. This event will always be logged at INFO level, since flows are only queued once.

  47. def recordRunning(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as RUNNING.

  48. def recordSkipped(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as SKIPPED.

    Records flow progress events with flow status as SKIPPED. For flows skipped because of upstream failures use recordSkippedOnUpStreamFailure function.

  49. def recordSkippedOnUpStreamFailure(flow: ResolvedFlow): Unit

    Records flow progress events with flow status as SKIPPED at WARN level, this version of record skipped should be used when the flow is skipped because of upstream flow failures.

  50. def recordStart(flowExecution: FlowExecution): Unit

    Records flow progress events with flow status as STARTING.

    Records flow progress events with flow status as STARTING. For batch flows in continuous mode, event will be logged at INFO if the recent flow run had failed otherwise the event will be logged at METRICS. All other cases will be logged at INFO.

  51. def recordStop(flow: ResolvedFlow, message: Option[String] = None, cause: Option[Throwable] = None): Unit

    Records flow progress events with flow status as STOPPED.

    Records flow progress events with flow status as STOPPED. This event will always be logged at INFO level, since flows wouldn't run after they are stopped.

  52. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  53. def toString(): String
    Definition Classes
    AnyRef → Any
  54. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  55. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  56. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  57. def withLogContext(context: Map[String, String])(body: => Unit): Unit
    Attributes
    protected
    Definition Classes
    Logging

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable]) @Deprecated
    Deprecated

    (Since version 9)

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Ungrouped