public class PeriodicGraphCheckpointer<VD,ED> extends Object implements Logging
Users should call PeriodicGraphCheckpointer.updateGraph()
when a new graph has been created,
before the graph has been materialized. After updating PeriodicGraphCheckpointer
, users are
responsible for materializing the graph to ensure that persisting and checkpointing actually
occur.
When PeriodicGraphCheckpointer.updateGraph()
is called, this does the following:
- Persist new graph (if not yet persisted), and put in queue of persisted graphs.
- Unpersist graphs from queue until there are at most 3 persisted graphs.
- If using checkpointing and the checkpoint interval has been reached,
- Checkpoint the new graph, and put in a queue of checkpointed graphs.
- Remove older checkpoints.
WARNINGS: - This class should NOT be copied (since copies may conflict on which Graphs should be checkpointed). - This class removes checkpoint files once later graphs have been checkpointed. However, references to the older graphs will still return isCheckpointed = true.
Example usage:
val (graph1, graph2, graph3, ...) = ...
val cp = new PeriodicGraphCheckpointer(graph1, dir, 2)
graph1.vertices.count(); graph1.edges.count()
// persisted: graph1
cp.updateGraph(graph2)
graph2.vertices.count(); graph2.edges.count()
// persisted: graph1, graph2
// checkpointed: graph2
cp.updateGraph(graph3)
graph3.vertices.count(); graph3.edges.count()
// persisted: graph1, graph2, graph3
// checkpointed: graph2
cp.updateGraph(graph4)
graph4.vertices.count(); graph4.edges.count()
// persisted: graph2, graph3, graph4
// checkpointed: graph4
cp.updateGraph(graph5)
graph5.vertices.count(); graph5.edges.count()
// persisted: graph3, graph4, graph5
// checkpointed: graph4
Constructor and Description |
---|
PeriodicGraphCheckpointer(Graph<VD,ED> currentGraph,
int checkpointInterval) |
Modifier and Type | Method and Description |
---|---|
int |
checkpointInterval() |
Graph<VD,ED> |
currentGraph() |
void |
deleteAllCheckpoints()
Call this at the end to delete any remaining checkpoint files.
|
void |
updateGraph(Graph<VD,ED> newGraph)
Update
currentGraph with a new graph. |
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning
public int checkpointInterval()
public void updateGraph(Graph<VD,ED> newGraph)
currentGraph
with a new graph. Handle persistence and checkpointing as needed.
Since this handles persistence and checkpointing, this should be called before the graph
has been materialized.
newGraph
- New graph created from previous graphs in the lineage.public void deleteAllCheckpoints()