Package org.apache.spark.streaming.util

Class WriteAheadLog

Object
org.apache.spark.streaming.util.WriteAheadLog
@DeveloperApi public abstract class WriteAheadLog extends Object
:: DeveloperApi :: This abstract class represents a write ahead log (aka journal) that is used by Spark Streaming to save the received data (by receivers) and associated metadata to a reliable storage, so that they can be recovered after driver failures. See the Spark documentation for more information on how to plug in your own custom implementation of a write ahead log.

  • Constructor Details

    • WriteAheadLog

      public WriteAheadLog()

  • Method Details

    • write

      public abstract WriteAheadLogRecordHandle write(ByteBuffer record, long time)
      Write the record to the log and return a record handle, which contains all the information necessary to read back the written record. The time is used to the index the record, such that it can be cleaned later. Note that implementations of this abstract class must ensure that the written data is durable and readable (using the record handle) by the time this function returns.

    • read

      public abstract ByteBuffer read(WriteAheadLogRecordHandle handle)
      Read a written record based on the given record handle.

    • readAll

      public abstract Iterator<ByteBuffer> readAll()
      Read and return an iterator of all the records that have been written but not yet cleaned up.

    • clean

      public abstract void clean(long threshTime, boolean waitForCompletion)
      Clean all the records that are older than the threshold time. It can wait for the completion of the deletion.

    • close

      public abstract void close()
      Close this log and release any resources. It must be idempotent.