Package org.apache.spark.streaming.util

Class WriteAheadLogUtils

Object

org.apache.spark.streaming.util.WriteAheadLogUtils

public class WriteAheadLogUtils
extends Object

A helper class with utility functions related to the WriteAheadLog interface

Constructor Summary

Constructors

Constructor	Description
WriteAheadLogUtils()

Method Summary

Modifier and Type	Method	Description
static WriteAheadLog	createLogForDriver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf)	Create a WriteAheadLog for the driver.
static WriteAheadLog	createLogForReceiver(SparkConf sparkConf, String fileWalLogDirectory, org.apache.hadoop.conf.Configuration fileWalHadoopConf)	Create a WriteAheadLog for the receiver.
static boolean	enableReceiverLog(SparkConf conf)
static long	getBatchingTimeout(SparkConf conf)	How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.
static int	getMaxFailures(SparkConf conf, boolean isDriver)
static int	getRollingIntervalSecs(SparkConf conf, boolean isDriver)
static boolean	isBatchingEnabled(SparkConf conf, boolean isDriver)
static org.apache.spark.internal.Logging.LogStringContext	LogStringContext(scala.StringContext sc)
static org.slf4j.Logger	org$apache$spark$internal$Logging$$log_()
static void	org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)
static boolean	shouldCloseFileAfterWrite(SparkConf conf, boolean isDriver)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WriteAheadLogUtils

public WriteAheadLogUtils()

Method Details

enableReceiverLog

public static boolean enableReceiverLog(SparkConf conf)

getRollingIntervalSecs

public static int getRollingIntervalSecs(SparkConf conf,
 boolean isDriver)

getMaxFailures

public static int getMaxFailures(SparkConf conf,
 boolean isDriver)

isBatchingEnabled

public static boolean isBatchingEnabled(SparkConf conf,
 boolean isDriver)

getBatchingTimeout

public static long getBatchingTimeout(SparkConf conf)

How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.

Parameters:

conf - (undocumented)

Returns:

(undocumented)

shouldCloseFileAfterWrite

public static boolean shouldCloseFileAfterWrite(SparkConf conf,
 boolean isDriver)

createLogForDriver

public static WriteAheadLog createLogForDriver(SparkConf sparkConf,
 String fileWalLogDirectory,
 org.apache.hadoop.conf.Configuration fileWalHadoopConf)

Create a WriteAheadLog for the driver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.

Parameters:

sparkConf - (undocumented)

fileWalLogDirectory - (undocumented)

fileWalHadoopConf - (undocumented)

Returns:

(undocumented)

createLogForReceiver

public static WriteAheadLog createLogForReceiver(SparkConf sparkConf,
 String fileWalLogDirectory,
 org.apache.hadoop.conf.Configuration fileWalHadoopConf)

Create a WriteAheadLog for the receiver. If configured with custom WAL class, it will try to create instance of that class, otherwise it will create the default FileBasedWriteAheadLog.

Parameters:

sparkConf - (undocumented)

fileWalLogDirectory - (undocumented)

fileWalHadoopConf - (undocumented)

Returns:

(undocumented)

org$apache$spark$internal$Logging$$log_

public static org.slf4j.Logger org$apache$spark$internal$Logging$$log_()

org$apache$spark$internal$Logging$$log__$eq

public static void org$apache$spark$internal$Logging$$log__$eq(org.slf4j.Logger x$1)

LogStringContext

public static org.apache.spark.internal.Logging.LogStringContext LogStringContext(scala.StringContext sc)