Package org.apache.spark.scheduler

Interface StructuredStreamingIdAwareSchedulerLogging

All Superinterfaces:
org.apache.spark.internal.Logging
public interface StructuredStreamingIdAwareSchedulerLogging extends org.apache.spark.internal.Logging
A logging trait for scheduler components where log messages should include structured streaming identifiers (query ID and batch ID).

Streaming execution sets these identifiers via SparkContext.setLocalProperty(java.lang.String, java.lang.String), which is thread-local. Scheduler code typically runs on a different thread (e.g. the task-scheduler-event-loop-worker), so getLocalProperty would not have the streaming context. This trait instead reads the identifiers from the task's Properties, which are propagated with the TaskSet across thread boundaries.

Mix this trait into any scheduler component that has access to task properties and needs streaming-aware log output.

  • Nested Class Summary

    Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

    org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    logDebug(org.apache.spark.internal.LogEntry entry)
     
    void
    logDebug(org.apache.spark.internal.LogEntry entry, Throwable t)
     
    void
    logDebug(scala.Function0<String> msg)
     
    void
    logDebug(scala.Function0<String> msg, Throwable t)
     
    void
    logError(org.apache.spark.internal.LogEntry entry)
     
    void
    logError(org.apache.spark.internal.LogEntry entry, Throwable t)
     
    void
    logError(scala.Function0<String> msg)
     
    void
    logError(scala.Function0<String> msg, Throwable t)
     
    void
    logInfo(org.apache.spark.internal.LogEntry entry)
     
    void
    logInfo(org.apache.spark.internal.LogEntry entry, Throwable t)
     
    void
    logInfo(scala.Function0<String> msg)
     
    void
    logInfo(scala.Function0<String> msg, Throwable t)
     
    void
    logTrace(org.apache.spark.internal.LogEntry entry)
     
    void
    logTrace(org.apache.spark.internal.LogEntry entry, Throwable t)
     
    void
    logTrace(scala.Function0<String> msg)
     
    void
    logTrace(scala.Function0<String> msg, Throwable t)
     
    void
    logWarning(org.apache.spark.internal.LogEntry entry)
     
    void
    logWarning(org.apache.spark.internal.LogEntry entry, Throwable t)
     
    void
    logWarning(scala.Function0<String> msg)
     
    void
    logWarning(scala.Function0<String> msg, Throwable t)
     
    Properties
    properties()
     
    boolean
    streamingIdAwareLoggingEnabled()
     
    int
    streamingQueryIdLength()
     

    Methods inherited from interface org.apache.spark.internal.Logging

    initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logName, LogStringContext, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

  • Method Details

    • properties

      Properties properties()

    • streamingIdAwareLoggingEnabled

      boolean streamingIdAwareLoggingEnabled()

    • streamingQueryIdLength

      int streamingQueryIdLength()

    • logInfo

      void logInfo(scala.Function0<String> msg)
      Specified by:
      logInfo in interface org.apache.spark.internal.Logging

    • logInfo

      void logInfo(org.apache.spark.internal.LogEntry entry)
      Specified by:
      logInfo in interface org.apache.spark.internal.Logging

    • logInfo

      void logInfo(scala.Function0<String> msg, Throwable t)
      Specified by:
      logInfo in interface org.apache.spark.internal.Logging

    • logInfo

      void logInfo(org.apache.spark.internal.LogEntry entry, Throwable t)
      Specified by:
      logInfo in interface org.apache.spark.internal.Logging

    • logWarning

      void logWarning(scala.Function0<String> msg)
      Specified by:
      logWarning in interface org.apache.spark.internal.Logging

    • logWarning

      void logWarning(org.apache.spark.internal.LogEntry entry)
      Specified by:
      logWarning in interface org.apache.spark.internal.Logging

    • logWarning

      void logWarning(scala.Function0<String> msg, Throwable t)
      Specified by:
      logWarning in interface org.apache.spark.internal.Logging

    • logWarning

      void logWarning(org.apache.spark.internal.LogEntry entry, Throwable t)
      Specified by:
      logWarning in interface org.apache.spark.internal.Logging

    • logDebug

      void logDebug(scala.Function0<String> msg)
      Specified by:
      logDebug in interface org.apache.spark.internal.Logging

    • logDebug

      void logDebug(org.apache.spark.internal.LogEntry entry)
      Specified by:
      logDebug in interface org.apache.spark.internal.Logging

    • logDebug

      void logDebug(scala.Function0<String> msg, Throwable t)
      Specified by:
      logDebug in interface org.apache.spark.internal.Logging

    • logDebug

      void logDebug(org.apache.spark.internal.LogEntry entry, Throwable t)
      Specified by:
      logDebug in interface org.apache.spark.internal.Logging

    • logError

      void logError(scala.Function0<String> msg)
      Specified by:
      logError in interface org.apache.spark.internal.Logging

    • logError

      void logError(org.apache.spark.internal.LogEntry entry)
      Specified by:
      logError in interface org.apache.spark.internal.Logging

    • logError

      void logError(scala.Function0<String> msg, Throwable t)
      Specified by:
      logError in interface org.apache.spark.internal.Logging

    • logError

      void logError(org.apache.spark.internal.LogEntry entry, Throwable t)
      Specified by:
      logError in interface org.apache.spark.internal.Logging

    • logTrace

      void logTrace(scala.Function0<String> msg)
      Specified by:
      logTrace in interface org.apache.spark.internal.Logging

    • logTrace

      void logTrace(org.apache.spark.internal.LogEntry entry)
      Specified by:
      logTrace in interface org.apache.spark.internal.Logging

    • logTrace

      void logTrace(scala.Function0<String> msg, Throwable t)
      Specified by:
      logTrace in interface org.apache.spark.internal.Logging

    • logTrace

      void logTrace(org.apache.spark.internal.LogEntry entry, Throwable t)
      Specified by:
      logTrace in interface org.apache.spark.internal.Logging