Class WriteAheadLogUtils

Object
org.apache.spark.streaming.util.WriteAheadLogUtils

public class WriteAheadLogUtils extends Object
A helper class with utility functions related to the WriteAheadLog interface
  • Constructor Details

    • WriteAheadLogUtils

      public WriteAheadLogUtils()
  • Method Details

    • enableReceiverLog

      public static boolean enableReceiverLog(SparkConf conf)
    • getRollingIntervalSecs

      public static int getRollingIntervalSecs(SparkConf conf, boolean isDriver)
    • getMaxFailures

      public static int getMaxFailures(SparkConf conf, boolean isDriver)
    • isBatchingEnabled

      public static boolean isBatchingEnabled(SparkConf conf, boolean isDriver)
    • getBatchingTimeout

      public static long getBatchingTimeout(SparkConf conf)
      How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.
      Parameters:
      conf - (undocumented)
      Returns:
      (undocumented)
    • shouldCloseFileAfterWrite

      public static boolean shouldCloseFileAfterWrite(SparkConf conf, boolean isDriver)
    • createLogForDriver

      public static WriteAheadLog createLogForDriver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf)
      Create a WriteAheadLog for the driver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.
      Parameters:
      sparkConf - (undocumented)
      fileWalLogDirectory - (undocumented)
      fileWalHadoopConf - (undocumented)
      Returns:
      (undocumented)
    • createLogForReceiver

      public static WriteAheadLog createLogForReceiver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf)
      Create a WriteAheadLog for the receiver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.
      Parameters:
      sparkConf - (undocumented)
      fileWalLogDirectory - (undocumented)
      fileWalHadoopConf - (undocumented)
      Returns:
      (undocumented)
    • org$apache$spark$internal$Logging$$log_

      public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()
    • org$apache$spark$internal$Logging$$log__$eq

      public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
    • LogStringContext

      public static org.apache.spark.internal.Logging.LogStringContext LogStringContext(scala.StringContext sc)