abstract class WriteAheadLog extends AnyRef
This abstract class represents a write ahead log (aka journal) that is used by Spark Streaming to save the received data (by receivers) and associated metadata to a reliable storage, so that they can be recovered after driver failures. See the Spark documentation for more information on how to plug in your own custom implementation of a write ahead log.
- Annotations
- @DeveloperApi()
- Source
- WriteAheadLog.java
- Alphabetic
- By Inheritance
- WriteAheadLog
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new WriteAheadLog()
Abstract Value Members
- abstract def clean(threshTime: Long, waitForCompletion: Boolean): Unit
Clean all the records that are older than the threshold time.
Clean all the records that are older than the threshold time. It can wait for the completion of the deletion.
- abstract def close(): Unit
Close this log and release any resources.
Close this log and release any resources. It must be idempotent.
- abstract def read(handle: WriteAheadLogRecordHandle): ByteBuffer
Read a written record based on the given record handle.
- abstract def readAll(): Iterator[ByteBuffer]
Read and return an iterator of all the records that have been written but not yet cleaned up.
- abstract def write(record: ByteBuffer, time: Long): WriteAheadLogRecordHandle
Write the record to the log and return a record handle, which contains all the information necessary to read back the written record.
Write the record to the log and return a record handle, which contains all the information necessary to read back the written record. The time is used to the index the record, such that it can be cleaned later. Note that implementations of this abstract class must ensure that the written data is durable and readable (using the record handle) by the time this function returns.