Package org.apache.spark.graphx.impl

Class EdgeRDDImpl<ED,VD>

Object

org.apache.spark.rdd.RDD<Edge<ED>>

org.apache.spark.graphx.EdgeRDD<ED>

org.apache.spark.graphx.impl.EdgeRDDImpl<ED,VD>

All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging

public class EdgeRDDImpl<ED,VD>
extends EdgeRDD<ED>

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Method Summary

Modifier and Type	Method	Description
EdgeRDDImpl<ED,VD>	cache()	Persists the edge partitions using targetStorageLevel, which defaults to MEMORY_ONLY.
void	checkpoint()	Mark this RDD for checkpointing.
Edge<ED>[]	collect()	Return an array that contains all of the elements in this RDD.
long	count()	The number of edges in the RDD.
EdgeRDDImpl<ED,VD>	filter(scala.Function1<EdgeTriplet<VD,ED>,Object> epred, scala.Function2<Object,VD,Object> vpred)
scala.Option<String>	getCheckpointFile()	Gets the name of the directory to which this RDD was checkpointed.
StorageLevel	getStorageLevel()	Get the RDD's current storage level, or StorageLevel.NONE if none is set.
<ED2, ED3> EdgeRDDImpl<ED3,VD>	innerJoin(EdgeRDD<ED2> other, scala.Function4<Object,Object,ED,ED2,ED3> f, scala.reflect.ClassTag<ED2> evidence$4, scala.reflect.ClassTag<ED3> evidence$5)	Inner joins this EdgeRDD with another EdgeRDD, assuming both are partitioned using the same PartitionStrategy.
boolean	isCheckpointed()	Return whether this RDD is checkpointed and materialized, either reliably or locally.
<ED2, VD2> EdgeRDDImpl<ED2,VD2>	mapEdgePartitions(scala.Function2<Object,org.apache.spark.graphx.impl.EdgePartition<ED,VD>,org.apache.spark.graphx.impl.EdgePartition<ED2,VD2>> f, scala.reflect.ClassTag<ED2> evidence$6, scala.reflect.ClassTag<VD2> evidence$7)
<ED2> EdgeRDDImpl<ED2,VD>	mapValues(scala.Function1<Edge<ED>,ED2> f, scala.reflect.ClassTag<ED2> evidence$3)	Map the values in an edge partitioning preserving the structure but changing the values.
scala.Option<Partitioner>	partitioner()	If partitionsRDD already has a partitioner, use it.
RDD<scala.Tuple2<Object,org.apache.spark.graphx.impl.EdgePartition<ED,VD>>>	partitionsRDD()
EdgeRDDImpl<ED,VD>	persist(StorageLevel newLevel)	Persists the edge partitions at the specified storage level, ignoring any existing target storage level.
EdgeRDDImpl<ED,VD>	reverse()	Reverse all the edges in this RDD.
EdgeRDDImpl<ED,VD>	setName(String _name)	Assign a name to this RDD
StorageLevel	targetStorageLevel()
EdgeRDDImpl<ED,VD>	unpersist(boolean blocking)	Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

Methods inherited from class org.apache.spark.graphx.EdgeRDD

compute, fromEdges

Methods inherited from class org.apache.spark.rdd.RDD

aggregate, barrier, cartesian, cleanShuffleDependencies, coalesce, collect, context, countApprox, countApproxDistinct, countApproxDistinct, countByValue, countByValueApprox, dependencies, distinct, distinct, doubleRDDToDoubleRDDFunctions, filter, first, flatMap, fold, foreach, foreachPartition, getNumPartitions, getResourceProfile, glom, groupBy, groupBy, groupBy, id, intersection, intersection, intersection, isEmpty, iterator, keyBy, localCheckpoint, map, mapPartitions, mapPartitionsWithEvaluator, mapPartitionsWithIndex, max, min, name, numericRDDToDoubleRDDFunctions, partitions, persist, pipe, pipe, pipe, preferredLocations, randomSplit, rddToAsyncRDDActions, rddToOrderedRDDFunctions, rddToPairRDDFunctions, rddToSequenceFileRDDFunctions, reduce, repartition, sample, saveAsObjectFile, saveAsTextFile, saveAsTextFile, sortBy, sparkContext, subtract, subtract, subtract, take, takeOrdered, takeSample, toDebugString, toJavaRDD, toLocalIterator, top, toString, treeAggregate, treeAggregate, treeReduce, union, withResources, zip, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitionsWithEvaluator, zipWithIndex, zipWithUniqueId

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logBasedOnLevel, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, MDC, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Method Details

partitionsRDD

public RDD<scala.Tuple2<Object,org.apache.spark.graphx.impl.EdgePartition<ED,VD>>> partitionsRDD()

targetStorageLevel

public StorageLevel targetStorageLevel()

setName

public EdgeRDDImpl<ED,VD> setName(String _name)

Description copied from class: RDD

Assign a name to this RDD

Overrides:

setName in class RDD<Edge<ED>>

partitioner

public scala.Option<Partitioner> partitioner()

If partitionsRDD already has a partitioner, use it. Otherwise assume that the PartitionIDs in partitionsRDD correspond to the actual partitions and create a new partitioner that allows co-partitioning with partitionsRDD.

Overrides:

partitioner in class RDD<Edge<ED>>

Returns:

(undocumented)

collect

public Edge<ED>[] collect()

Description copied from class: RDD

Return an array that contains all of the elements in this RDD.

Overrides:

collect in class RDD<Edge<ED>>

Returns:

(undocumented)

persist

public EdgeRDDImpl<ED,VD> persist(StorageLevel newLevel)

Persists the edge partitions at the specified storage level, ignoring any existing target storage level.

Overrides:

persist in class RDD<Edge<ED>>

Parameters:

newLevel - (undocumented)

Returns:

(undocumented)

unpersist

public EdgeRDDImpl<ED,VD> unpersist(boolean blocking)

Description copied from class: RDD

Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

Overrides:

unpersist in class RDD<Edge<ED>>

Parameters:

blocking - Whether to block until all blocks are deleted (default: false)

Returns:

This RDD.

cache

public EdgeRDDImpl<ED,VD> cache()

Persists the edge partitions using targetStorageLevel, which defaults to MEMORY_ONLY.

Overrides:

cache in class RDD<Edge<ED>>

Returns:

(undocumented)

getStorageLevel

public StorageLevel getStorageLevel()

Description copied from class: RDD

Get the RDD's current storage level, or StorageLevel.NONE if none is set.

Overrides:

getStorageLevel in class RDD<Edge<ED>>

checkpoint

public void checkpoint()

Description copied from class: RDD

Mark this RDD for checkpointing. It will be saved to a file inside the checkpoint directory set with SparkContext#setCheckpointDir and all references to its parent RDDs will be removed. This function must be called before any job has been executed on this RDD. It is strongly recommended that this RDD is persisted in memory, otherwise saving it on a file will require recomputation.

The data is only checkpointed when doCheckpoint() is called, and this only happens at the end of the first action execution on this RDD. The final data that is checkpointed after the first action may be different from the data that was used during the action, due to non-determinism of the underlying operation and retries. If the purpose of the checkpoint is to achieve saving a deterministic snapshot of the data, an eager action may need to be called first on the RDD to trigger the checkpoint.

Overrides:

checkpoint in class RDD<Edge<ED>>

isCheckpointed

public boolean isCheckpointed()

Description copied from class: RDD

Return whether this RDD is checkpointed and materialized, either reliably or locally.

Overrides:

isCheckpointed in class RDD<Edge<ED>>

Returns:

(undocumented)

getCheckpointFile

public scala.Option<String> getCheckpointFile()

Description copied from class: RDD

Gets the name of the directory to which this RDD was checkpointed. This is not defined if the RDD is checkpointed locally.

Overrides:

getCheckpointFile in class RDD<Edge<ED>>

Returns:

(undocumented)

count

public long count()

The number of edges in the RDD.

Overrides:

count in class RDD<Edge<ED>>

Returns:

(undocumented)

mapValues

public <ED2> EdgeRDDImpl<ED2,VD> mapValues(scala.Function1<Edge<ED>,ED2> f,
 scala.reflect.ClassTag<ED2> evidence$3)

Description copied from class: EdgeRDD

Map the values in an edge partitioning preserving the structure but changing the values.

Specified by:

mapValues in class EdgeRDD<ED>

Parameters:

f - the function from an edge to a new edge value

evidence$3 - (undocumented)

Returns:

a new EdgeRDD containing the new edge values

reverse

public EdgeRDDImpl<ED,VD> reverse()

Description copied from class: EdgeRDD

Reverse all the edges in this RDD.

Specified by:

reverse in class EdgeRDD<ED>

Returns:

a new EdgeRDD containing all the edges reversed

filter

public EdgeRDDImpl<ED,VD> filter(scala.Function1<EdgeTriplet<VD,ED>,Object> epred,
 scala.Function2<Object,VD,Object> vpred)

innerJoin

public <ED2,
ED3> EdgeRDDImpl<ED3,VD> innerJoin(EdgeRDD<ED2> other,
 scala.Function4<Object,Object,ED,ED2,ED3> f,
 scala.reflect.ClassTag<ED2> evidence$4,
 scala.reflect.ClassTag<ED3> evidence$5)

Description copied from class: EdgeRDD

Inner joins this EdgeRDD with another EdgeRDD, assuming both are partitioned using the same PartitionStrategy.

Specified by:

innerJoin in class EdgeRDD<ED>

Parameters:

other - the EdgeRDD to join with

f - the join function applied to corresponding values of this and other

evidence$4 - (undocumented)

evidence$5 - (undocumented)

Returns:

a new EdgeRDD containing only edges that appear in both this and other, with values supplied by f

mapEdgePartitions

public <ED2,
VD2> EdgeRDDImpl<ED2,VD2> mapEdgePartitions(scala.Function2<Object,org.apache.spark.graphx.impl.EdgePartition<ED,VD>,org.apache.spark.graphx.impl.EdgePartition<ED2,VD2>> f,
 scala.reflect.ClassTag<ED2> evidence$6,
 scala.reflect.ClassTag<VD2> evidence$7)