public class PeriodicGraphCheckpointer<VD,ED> extends Object implements Logging
Users should call
PeriodicGraphCheckpointer.updateGraph() when a new graph has been created,
before the graph has been materialized. After updating
PeriodicGraphCheckpointer, users are
responsible for materializing the graph to ensure that persisting and checkpointing actually
PeriodicGraphCheckpointer.updateGraph() is called, this does the following:
- Persist new graph (if not yet persisted), and put in queue of persisted graphs.
- Unpersist graphs from queue until there are at most 3 persisted graphs.
- If using checkpointing and the checkpoint interval has been reached,
- Checkpoint the new graph, and put in a queue of checkpointed graphs.
- Remove older checkpoints.
WARNINGS: - This class should NOT be copied (since copies may conflict on which Graphs should be checkpointed). - This class removes checkpoint files once later graphs have been checkpointed. However, references to the older graphs will still return isCheckpointed = true.
val (graph1, graph2, graph3, ...) = ... val cp = new PeriodicGraphCheckpointer(graph1, dir, 2) graph1.vertices.count(); graph1.edges.count() // persisted: graph1 cp.updateGraph(graph2) graph2.vertices.count(); graph2.edges.count() // persisted: graph1, graph2 // checkpointed: graph2 cp.updateGraph(graph3) graph3.vertices.count(); graph3.edges.count() // persisted: graph1, graph2, graph3 // checkpointed: graph2 cp.updateGraph(graph4) graph4.vertices.count(); graph4.edges.count() // persisted: graph2, graph3, graph4 // checkpointed: graph4 cp.updateGraph(graph5) graph5.vertices.count(); graph5.edges.count() // persisted: graph3, graph4, graph5 // checkpointed: graph4
|Constructor and Description|
|Modifier and Type||Method and Description|
Call this at the end to delete any remaining checkpoint files.
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
public int checkpointInterval()
currentGraphwith a new graph. Handle persistence and checkpointing as needed. Since this handles persistence and checkpointing, this should be called before the graph has been materialized.
newGraph- New graph created from previous graphs in the lineage.
public void deleteAllCheckpoints()