Package org.apache.spark.sql.pipelines.logging

Class FlowProgressEventLogger

Object
org.apache.spark.sql.pipelines.logging.FlowProgressEventLogger
All Implemented Interfaces:
org.apache.spark.internal.Logging
public class FlowProgressEventLogger extends Object implements org.apache.spark.internal.Logging
This class should be used for all flow progress events logging, it controls the level at which events are logged. It uses execution mode, flow name and previous flow statuses to infer the level at which an event is to be logged. Below is a more details description of how flow progress events for batch/streaming flows will be logged:

For batch & streaming flows in triggered execution mode: - All flow progress events other than errors/warnings will be logged at INFO level (including flow progress events with metrics) and error/warning messages will be logged at their level.

param: eventBuffer Event log to log the flow progress events.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Constructor Summary

    Constructors
    Constructor
    Description
    FlowProgressEventLogger(PipelineRunEventBuffer eventBuffer)
     

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    recordCompletion(ResolvedFlow flow)
    Records flow progress events with flow status as COMPLETED.
    void
    recordExcluded(ResolvedFlow flow)
    Records flow progress events with flow status as EXCLUDED at INFO level.
    void
    recordFailed(ResolutionCompletedFlow flow, Throwable exception, boolean logAsWarn, scala.Option<String> messageOpt)
    Records flow progress events with failure flow status.
    void
    recordIdle(ResolvedFlow flow)
    Records flow progress events with flow status as IDLE.
    void
    recordPlanningForBatchFlow(ResolvedFlow batchFlow)
    Records flow progress events with flow status as PLANNING for batch flows.
    void
    recordQueued(ResolvedFlow flow)
    Records flow progress events with flow status as QUEUED.
    void
    recordRunning(ResolvedFlow flow)
    Records flow progress events with flow status as RUNNING.
    void
    recordSkipped(ResolvedFlow flow)
    Records flow progress events with flow status as SKIPPED.
    void
    recordSkippedOnUpStreamFailure(ResolvedFlow flow)
    Records flow progress events with flow status as SKIPPED at WARN level, this version of record skipped should be used when the flow is skipped because of upstream flow failures.
    void
    recordStart(FlowExecution flowExecution)
    Records flow progress events with flow status as STARTING.
    void
    recordStop(ResolvedFlow flow, scala.Option<String> message, scala.Option<Throwable> cause)
    Records flow progress events with flow status as STOPPED.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Constructor Details

  • Method Details

    • recordCompletion

      public void recordCompletion(ResolvedFlow flow)
      Records flow progress events with flow status as COMPLETED. For batch flows in continuous mode, events will be logged at METRICS since a completed status is always preceded by running status.

      Note that flow complete events for batch flows are expected to contain quality stats where as for streaming flows quality stats are not expected and hence not added to the flow progress event.

      Parameters:
      flow - (undocumented)

    • recordExcluded

      public void recordExcluded(ResolvedFlow flow)
      Records flow progress events with flow status as EXCLUDED at INFO level.

    • recordFailed

      public void recordFailed(ResolutionCompletedFlow flow, Throwable exception, boolean logAsWarn, scala.Option<String> messageOpt)
      Records flow progress events with failure flow status. By default failed flow progress events are logged at ERROR level, logAsWarn serve as a way to log the event as a WARN.
      Parameters:
      flow - (undocumented)
      exception - (undocumented)
      logAsWarn - (undocumented)
      messageOpt - (undocumented)

    • recordIdle

      public void recordIdle(ResolvedFlow flow)
      Records flow progress events with flow status as IDLE.

    • recordPlanningForBatchFlow

      public void recordPlanningForBatchFlow(ResolvedFlow batchFlow)
      Records flow progress events with flow status as PLANNING for batch flows.
      Parameters:
      batchFlow - (undocumented)

    • recordQueued

      public void recordQueued(ResolvedFlow flow)
      Records flow progress events with flow status as QUEUED. This event will always be logged at INFO level, since flows are only queued once.
      Parameters:
      flow - (undocumented)

    • recordRunning

      public void recordRunning(ResolvedFlow flow)
      Records flow progress events with flow status as RUNNING.

    • recordSkipped

      public void recordSkipped(ResolvedFlow flow)
      Records flow progress events with flow status as SKIPPED. For flows skipped because of upstream failures use recordSkippedOnUpStreamFailure(org.apache.spark.sql.pipelines.graph.ResolvedFlow) function.
      Parameters:
      flow - (undocumented)

    • recordSkippedOnUpStreamFailure

      public void recordSkippedOnUpStreamFailure(ResolvedFlow flow)
      Records flow progress events with flow status as SKIPPED at WARN level, this version of record skipped should be used when the flow is skipped because of upstream flow failures.
      Parameters:
      flow - (undocumented)

    • recordStart

      public void recordStart(FlowExecution flowExecution)
      Records flow progress events with flow status as STARTING. For batch flows in continuous mode, event will be logged at INFO if the recent flow run had failed otherwise the event will be logged at METRICS. All other cases will be logged at INFO.
      Parameters:
      flowExecution - (undocumented)

    • recordStop

      public void recordStop(ResolvedFlow flow, scala.Option<String> message, scala.Option<Throwable> cause)
      Records flow progress events with flow status as STOPPED. This event will always be logged at INFO level, since flows wouldn't run after they are stopped.
      Parameters:
      flow - (undocumented)
      message - (undocumented)
      cause - (undocumented)