org.apache.spark

Class ContextCleaner

Object

org.apache.spark.ContextCleaner

All Implemented Interfaces:

Logging

public class ContextCleaner
extends Object
implements Logging

An asynchronous cleaner for RDD, shuffle, and broadcast state.

This maintains a weak reference for each RDD, ShuffleDependency, and Broadcast of interest, to be processed when the associated object goes out of scope of the application. Actual cleanup is performed in a separate daemon thread.

Constructor Summary

Constructors

Constructor and Description
ContextCleaner(SparkContext sc)

Method Summary

Methods

Modifier and Type	Method and Description
void	attachListener(CleanerListener listener) Attach a listener object to get information of when objects are cleaned.
void	doCleanupBroadcast(long broadcastId, boolean blocking) Perform broadcast cleanup.
void	doCleanupRDD(int rddId, boolean blocking) Perform RDD cleanup.
void	doCleanupShuffle(int shuffleId, boolean blocking) Perform shuffle cleanup, asynchronously.
<T> void	registerBroadcastForCleanup(Broadcast<T> broadcast) Register a Broadcast for cleanup when it is garbage collected.
void	registerRDDForCleanup(RDD<?> rdd) Register a RDD for cleanup when it is garbage collected.
void	registerShuffleForCleanup(ShuffleDependency<?,?,?> shuffleDependency) Register a ShuffleDependency for cleanup when it is garbage collected.
void	start() Start the cleaner.
void	stop() Stop the cleaning thread and wait until the thread has finished running its current task.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ContextCleaner

public ContextCleaner(SparkContext sc)

Method Detail

attachListener

public void attachListener(CleanerListener listener)

Attach a listener object to get information of when objects are cleaned.

start

public void start()

Start the cleaner.

stop

public void stop()

Stop the cleaning thread and wait until the thread has finished running its current task.

registerRDDForCleanup

public void registerRDDForCleanup(RDD<?> rdd)

Register a RDD for cleanup when it is garbage collected.

registerShuffleForCleanup

public void registerShuffleForCleanup(ShuffleDependency<?,?,?> shuffleDependency)

Register a ShuffleDependency for cleanup when it is garbage collected.

registerBroadcastForCleanup

public <T> void registerBroadcastForCleanup(Broadcast<T> broadcast)

Register a Broadcast for cleanup when it is garbage collected.

doCleanupRDD

public void doCleanupRDD(int rddId,
                boolean blocking)

Perform RDD cleanup.

doCleanupShuffle

public void doCleanupShuffle(int shuffleId,
                    boolean blocking)

Perform shuffle cleanup, asynchronously.

doCleanupBroadcast

public void doCleanupBroadcast(long broadcastId,
                      boolean blocking)

Perform broadcast cleanup.