public class MemoryStore extends BlockStore
Constructor and Description |
---|
MemoryStore(BlockManager blockManager,
long maxMemory) |
Modifier and Type | Method and Description |
---|---|
void |
clear() |
boolean |
contains(BlockId blockId) |
long |
currentUnrollMemory()
Return the amount of memory currently occupied for unrolling blocks across all threads.
|
long |
currentUnrollMemoryForThisThread()
Return the amount of memory currently occupied for unrolling blocks by this thread.
|
long |
freeMemory()
Free memory not occupied by existing blocks.
|
scala.Option<java.nio.ByteBuffer> |
getBytes(BlockId blockId) |
long |
getSize(BlockId blockId)
Return the size of a block in bytes.
|
scala.Option<scala.collection.Iterator<Object>> |
getValues(BlockId blockId) |
void |
logMemoryUsage()
Log information about current memory usage.
|
void |
logUnrollFailureMessage(BlockId blockId,
long finalVectorSize)
Log a warning for failing to unroll a block.
|
int |
numThreadsUnrolling()
Return the number of threads currently unrolling blocks.
|
PutResult |
putArray(BlockId blockId,
Object[] values,
StorageLevel level,
boolean returnValues) |
PutResult |
putBytes(BlockId blockId,
java.nio.ByteBuffer _bytes,
StorageLevel level) |
PutResult |
putIterator(BlockId blockId,
scala.collection.Iterator<Object> values,
StorageLevel level,
boolean returnValues)
Put in a block and, possibly, also return its content as either bytes or another Iterator.
|
PutResult |
putIterator(BlockId blockId,
scala.collection.Iterator<Object> values,
StorageLevel level,
boolean returnValues,
boolean allowPersistToDisk)
Attempt to put the given block in memory store.
|
void |
releaseUnrollMemoryForThisThread(long memory)
Release memory used by this thread for unrolling blocks.
|
boolean |
remove(BlockId blockId)
Remove a block, if it exists.
|
boolean |
reserveUnrollMemoryForThisThread(long memory)
Reserve additional memory for unrolling blocks used by this thread.
|
scala.util.Either<Object[],scala.collection.Iterator<Object>> |
unrollSafely(BlockId blockId,
scala.collection.Iterator<Object> values,
scala.collection.mutable.ArrayBuffer<scala.Tuple2<BlockId,BlockStatus>> droppedBlocks)
Unroll the given block in memory safely.
|
blockManager
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
public MemoryStore(BlockManager blockManager, long maxMemory)
public long freeMemory()
public long getSize(BlockId blockId)
BlockStore
getSize
in class BlockStore
public PutResult putBytes(BlockId blockId, java.nio.ByteBuffer _bytes, StorageLevel level)
putBytes
in class BlockStore
public PutResult putArray(BlockId blockId, Object[] values, StorageLevel level, boolean returnValues)
putArray
in class BlockStore
public PutResult putIterator(BlockId blockId, scala.collection.Iterator<Object> values, StorageLevel level, boolean returnValues)
BlockStore
putIterator
in class BlockStore
public PutResult putIterator(BlockId blockId, scala.collection.Iterator<Object> values, StorageLevel level, boolean returnValues, boolean allowPersistToDisk)
There may not be enough space to fully unroll the iterator in memory, in which case we
optionally drop the values to disk if
(1) the block's storage level specifies useDisk, and
(2) allowPersistToDisk
is true.
One scenario in which allowPersistToDisk
is false is when the BlockManager reads a block
back from disk and attempts to cache it in memory. In this case, we should not persist the
block back on disk again, as it is already in disk store.
public scala.Option<java.nio.ByteBuffer> getBytes(BlockId blockId)
getBytes
in class BlockStore
public scala.Option<scala.collection.Iterator<Object>> getValues(BlockId blockId)
getValues
in class BlockStore
public boolean remove(BlockId blockId)
BlockStore
remove
in class BlockStore
blockId
- the block to remove.public void clear()
clear
in class BlockStore
public scala.util.Either<Object[],scala.collection.Iterator<Object>> unrollSafely(BlockId blockId, scala.collection.Iterator<Object> values, scala.collection.mutable.ArrayBuffer<scala.Tuple2<BlockId,BlockStatus>> droppedBlocks)
The safety of this operation refers to avoiding potential OOM exceptions caused by unrolling the entirety of the block in memory at once. This is achieved by periodically checking whether the memory restrictions for unrolling blocks are still satisfied, stopping immediately if not. This check is a safeguard against the scenario in which there is not enough free memory to accommodate the entirety of a single block.
This method returns either an array with the contents of the entire block or an iterator containing the values of the block (if the array would have exceeded available memory).
public boolean contains(BlockId blockId)
contains
in class BlockStore
public boolean reserveUnrollMemoryForThisThread(long memory)
public void releaseUnrollMemoryForThisThread(long memory)
public long currentUnrollMemory()
public long currentUnrollMemoryForThisThread()
public int numThreadsUnrolling()
public void logMemoryUsage()
public void logUnrollFailureMessage(BlockId blockId, long finalVectorSize)
blockId
- ID of the block we are trying to unroll.finalVectorSize
- Final size of the vector before unrolling failed.