Package org.apache.spark.graphx.impl

Class VertexRDDImpl<VD>

Object

org.apache.spark.rdd.RDD<scala.Tuple2<Object,VD>>

org.apache.spark.graphx.VertexRDD<VD>

org.apache.spark.graphx.impl.VertexRDDImpl<VD>

All Implemented Interfaces:

Serializable, org.apache.spark.internal.Logging

public class VertexRDDImpl<VD>
extends VertexRDD<VD>

See Also:

• Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.spark.internal.Logging

org.apache.spark.internal.Logging.LogStringContext, org.apache.spark.internal.Logging.SparkShellLoggingFilter

Method Summary

Modifier and Type	Method	Description
<VD2> VertexRDD<VD2>	aggregateUsingIndex(RDD<scala.Tuple2<Object,VD2>> messages, scala.Function2<VD2,VD2,VD2> reduceFunc, scala.reflect.ClassTag<VD2> evidence$12)	Aggregates vertices in messages that have the same ids using reduceFunc, returning a VertexRDD co-indexed with this.
VertexRDDImpl<VD>	cache()	Persists the vertex partitions at targetStorageLevel, which defaults to MEMORY_ONLY.
void	checkpoint()	Mark this RDD for checkpointing.
long	count()	The number of vertices in the RDD.
VertexRDD<VD>	diff(VertexRDD<VD> other)	For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other.
VertexRDD<VD>	diff(RDD<scala.Tuple2<Object,VD>> other)	For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other.
scala.Option<String>	getCheckpointFile()	Gets the name of the directory to which this RDD was checkpointed.
StorageLevel	getStorageLevel()	Get the RDD's current storage level, or StorageLevel.NONE if none is set.
<U, VD2> VertexRDD<VD2>	innerJoin(RDD<scala.Tuple2<Object,U>> other, scala.Function3<Object,VD,U,VD2> f, scala.reflect.ClassTag<U> evidence$10, scala.reflect.ClassTag<VD2> evidence$11)	Inner joins this VertexRDD with an RDD containing vertex attribute pairs.
<U, VD2> VertexRDD<VD2>	innerZipJoin(VertexRDD<U> other, scala.Function3<Object,VD,U,VD2> f, scala.reflect.ClassTag<U> evidence$8, scala.reflect.ClassTag<VD2> evidence$9)	Efficiently inner joins this VertexRDD with another VertexRDD sharing the same index.
boolean	isCheckpointed()	Return whether this RDD is checkpointed and materialized, either reliably or locally.
<VD2, VD3> VertexRDD<VD3>	leftJoin(RDD<scala.Tuple2<Object,VD2>> other, scala.Function3<Object,VD,scala.Option<VD2>,VD3> f, scala.reflect.ClassTag<VD2> evidence$6, scala.reflect.ClassTag<VD3> evidence$7)	Left joins this VertexRDD with an RDD containing vertex attribute pairs.
<VD2, VD3> VertexRDD<VD3>	leftZipJoin(VertexRDD<VD2> other, scala.Function3<Object,VD,scala.Option<VD2>,VD3> f, scala.reflect.ClassTag<VD2> evidence$4, scala.reflect.ClassTag<VD3> evidence$5)	Left joins this RDD with another VertexRDD with the same index.
<VD2> VertexRDD<VD2>	mapValues(scala.Function1<VD,VD2> f, scala.reflect.ClassTag<VD2> evidence$2)	Maps each vertex attribute, preserving the index.
<VD2> VertexRDD<VD2>	mapValues(scala.Function2<Object,VD,VD2> f, scala.reflect.ClassTag<VD2> evidence$3)	Maps each vertex attribute, additionally supplying the vertex ID.
VertexRDD<VD>	minus(VertexRDD<VD> other)	For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.
VertexRDD<VD>	minus(RDD<scala.Tuple2<Object,VD>> other)	For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.
scala.Option<Partitioner>	partitioner()	Optionally overridden by subclasses to specify how they are partitioned.
RDD<org.apache.spark.graphx.impl.ShippableVertexPartition<VD>>	partitionsRDD()
VertexRDDImpl<VD>	persist(StorageLevel newLevel)	Persists the vertex partitions at the specified storage level, ignoring any existing target storage level.
VertexRDD<VD>	reindex()	Construct a new VertexRDD that is indexed by only the visible vertices.
VertexRDD<VD>	reverseRoutingTables()	Returns a new VertexRDD reflecting a reversal of all edge directions in the corresponding EdgeRDD.
VertexRDDImpl<VD>	setName(String _name)	Assign a name to this RDD
StorageLevel	targetStorageLevel()
VertexRDDImpl<VD>	unpersist(boolean blocking)	Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
VertexRDD<VD>	withEdges(EdgeRDD<?> edges)	Prepares this VertexRDD for efficient joins with the given EdgeRDD.

Methods inherited from class org.apache.spark.graphx.VertexRDD

apply, apply, apply, compute, filter, fromEdges

Methods inherited from class org.apache.spark.rdd.RDD

aggregate, barrier, cartesian, cleanShuffleDependencies, coalesce, collect, collect, context, countApprox, countApproxDistinct, countApproxDistinct, countByValue, countByValueApprox, dependencies, distinct, distinct, doubleRDDToDoubleRDDFunctions, first, flatMap, fold, foreach, foreachPartition, getNumPartitions, getResourceProfile, glom, groupBy, groupBy, groupBy, id, intersection, intersection, intersection, isEmpty, iterator, keyBy, localCheckpoint, map, mapPartitions, mapPartitionsWithEvaluator, mapPartitionsWithIndex, max, min, name, numericRDDToDoubleRDDFunctions, partitions, persist, pipe, pipe, pipe, preferredLocations, randomSplit, rddToAsyncRDDActions, rddToOrderedRDDFunctions, rddToPairRDDFunctions, rddToSequenceFileRDDFunctions, reduce, repartition, sample, saveAsObjectFile, saveAsTextFile, saveAsTextFile, sortBy, sparkContext, subtract, subtract, subtract, take, takeOrdered, takeSample, toDebugString, toJavaRDD, toLocalIterator, top, toString, treeAggregate, treeAggregate, treeReduce, union, withResources, zip, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitions, zipPartitionsWithEvaluator, zipWithIndex, zipWithUniqueId

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.spark.internal.Logging

initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, isTraceEnabled, log, logDebug, logDebug, logDebug, logDebug, logError, logError, logError, logError, logInfo, logInfo, logInfo, logInfo, logName, LogStringContext, logTrace, logTrace, logTrace, logTrace, logWarning, logWarning, logWarning, logWarning, org$apache$spark$internal$Logging$$log_, org$apache$spark$internal$Logging$$log__$eq, withLogContext

Method Details

partitionsRDD

public RDD<org.apache.spark.graphx.impl.ShippableVertexPartition<VD>> partitionsRDD()

targetStorageLevel

public StorageLevel targetStorageLevel()

reindex

public VertexRDD<VD> reindex()

Description copied from class: VertexRDD

Construct a new VertexRDD that is indexed by only the visible vertices. The resulting VertexRDD will be based on a different index and can no longer be quickly joined with this RDD.

Specified by:

reindex in class VertexRDD<VD>

Returns:

(undocumented)

partitioner

public scala.Option<Partitioner> partitioner()

Description copied from class: RDD

Optionally overridden by subclasses to specify how they are partitioned.

Overrides:

partitioner in class RDD<scala.Tuple2<Object,VD>>

setName

public VertexRDDImpl<VD> setName(String _name)

Description copied from class: RDD

Assign a name to this RDD

Overrides:

setName in class RDD<scala.Tuple2<Object,VD>>

persist

public VertexRDDImpl<VD> persist(StorageLevel newLevel)

Persists the vertex partitions at the specified storage level, ignoring any existing target storage level.

Overrides:

persist in class RDD<scala.Tuple2<Object,VD>>

Parameters:

newLevel - (undocumented)

Returns:

(undocumented)

unpersist

public VertexRDDImpl<VD> unpersist(boolean blocking)

Description copied from class: RDD

Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.

Overrides:

unpersist in class RDD<scala.Tuple2<Object,VD>>

Parameters:

blocking - Whether to block until all blocks are deleted (default: false)

Returns:

This RDD.

cache

public VertexRDDImpl<VD> cache()

Persists the vertex partitions at targetStorageLevel, which defaults to MEMORY_ONLY.

Overrides:

cache in class RDD<scala.Tuple2<Object,VD>>

Returns:

(undocumented)

getStorageLevel

public StorageLevel getStorageLevel()

Description copied from class: RDD

Get the RDD's current storage level, or StorageLevel.NONE if none is set.

Overrides:

getStorageLevel in class RDD<scala.Tuple2<Object,VD>>

checkpoint

public void checkpoint()

Description copied from class: RDD

Mark this RDD for checkpointing. It will be saved to a file inside the checkpoint directory set with SparkContext#setCheckpointDir and all references to its parent RDDs will be removed. This function must be called before any job has been executed on this RDD. It is strongly recommended that this RDD is persisted in memory, otherwise saving it on a file will require recomputation.

The data is only checkpointed when doCheckpoint() is called, and this only happens at the end of the first action execution on this RDD. The final data that is checkpointed after the first action may be different from the data that was used during the action, due to non-determinism of the underlying operation and retries. If the purpose of the checkpoint is to achieve saving a deterministic snapshot of the data, an eager action may need to be called first on the RDD to trigger the checkpoint.

Overrides:

checkpoint in class RDD<scala.Tuple2<Object,VD>>

isCheckpointed

public boolean isCheckpointed()

Description copied from class: RDD

Return whether this RDD is checkpointed and materialized, either reliably or locally.

Overrides:

isCheckpointed in class RDD<scala.Tuple2<Object,VD>>

Returns:

(undocumented)

getCheckpointFile

public scala.Option<String> getCheckpointFile()

Description copied from class: RDD

Gets the name of the directory to which this RDD was checkpointed. This is not defined if the RDD is checkpointed locally.

Overrides:

getCheckpointFile in class RDD<scala.Tuple2<Object,VD>>

Returns:

(undocumented)

count

public long count()

The number of vertices in the RDD.

Overrides:

count in class RDD<scala.Tuple2<Object,VD>>

Returns:

(undocumented)

mapValues

public <VD2> VertexRDD<VD2> mapValues(scala.Function1<VD,VD2> f,
 scala.reflect.ClassTag<VD2> evidence$2)

Description copied from class: VertexRDD

Maps each vertex attribute, preserving the index.

Specified by:

mapValues in class VertexRDD<VD>

Parameters:

f - the function applied to each value in the RDD

evidence$2 - (undocumented)

Returns:

a new VertexRDD with values obtained by applying f to each of the entries in the original VertexRDD

mapValues

public <VD2> VertexRDD<VD2> mapValues(scala.Function2<Object,VD,VD2> f,
 scala.reflect.ClassTag<VD2> evidence$3)

Description copied from class: VertexRDD

Maps each vertex attribute, additionally supplying the vertex ID.

Specified by:

mapValues in class VertexRDD<VD>

Parameters:

f - the function applied to each ID-value pair in the RDD

evidence$3 - (undocumented)

Returns:

a new VertexRDD with values obtained by applying f to each of the entries in the original VertexRDD. The resulting VertexRDD retains the same index.

minus

public VertexRDD<VD> minus(RDD<scala.Tuple2<Object,VD>> other)

Description copied from class: VertexRDD

For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.

Specified by:

minus in class VertexRDD<VD>

Parameters:

other - an RDD to run the set operation against

Returns:

(undocumented)

minus

public VertexRDD<VD> minus(VertexRDD<VD> other)

Description copied from class: VertexRDD

For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.

Specified by:

minus in class VertexRDD<VD>

Parameters:

other - a VertexRDD to run the set operation against

Returns:

(undocumented)

diff

public VertexRDD<VD> diff(RDD<scala.Tuple2<Object,VD>> other)

Description copied from class: VertexRDD

For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other. This is only guaranteed to work if the VertexRDDs share a common ancestor.

Specified by:

diff in class VertexRDD<VD>

Parameters:

other - the other RDD[(VertexId, VD)] with which to diff against.

Returns:

(undocumented)

diff

public VertexRDD<VD> diff(VertexRDD<VD> other)

Description copied from class: VertexRDD

For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other. This is only guaranteed to work if the VertexRDDs share a common ancestor.

Specified by:

diff in class VertexRDD<VD>

Parameters:

other - the other VertexRDD with which to diff against.

Returns:

(undocumented)

leftZipJoin

public <VD2,
VD3> VertexRDD<VD3> leftZipJoin(VertexRDD<VD2> other,
 scala.Function3<Object,VD,scala.Option<VD2>,VD3> f,
 scala.reflect.ClassTag<VD2> evidence$4,
 scala.reflect.ClassTag<VD3> evidence$5)

Description copied from class: VertexRDD

Left joins this RDD with another VertexRDD with the same index. This function will fail if both VertexRDDs do not share the same index. The resulting vertex set contains an entry for each vertex in this. If other is missing any vertex in this VertexRDD, f is passed None.

Specified by:

leftZipJoin in class VertexRDD<VD>

Parameters:

other - the other VertexRDD with which to join.

f - the function mapping a vertex id and its attributes in this and the other vertex set to a new vertex attribute.

evidence$4 - (undocumented)

evidence$5 - (undocumented)

Returns:

a VertexRDD containing the results of f

leftJoin

public <VD2,
VD3> VertexRDD<VD3> leftJoin(RDD<scala.Tuple2<Object,VD2>> other,
 scala.Function3<Object,VD,scala.Option<VD2>,VD3> f,
 scala.reflect.ClassTag<VD2> evidence$6,
 scala.reflect.ClassTag<VD3> evidence$7)

Description copied from class: VertexRDD

Left joins this VertexRDD with an RDD containing vertex attribute pairs. If the other RDD is backed by a VertexRDD with the same index then the efficient VertexRDD.<VD2,VD3>leftZipJoin(org.apache.spark.graphx.VertexRDD<VD2>,scala.Function3<java.lang.Object,VD,scala.Option<VD2>,VD3>,scala.reflect.ClassTag<VD2>,scala.reflect.ClassTag<VD3>) implementation is used. The resulting VertexRDD contains an entry for each vertex in this. If other is missing any vertex in this VertexRDD, f is passed None. If there are duplicates, the vertex is picked arbitrarily.

Specified by:

leftJoin in class VertexRDD<VD>

Parameters:

other - the other VertexRDD with which to join

f - the function mapping a vertex id and its attributes in this and the other vertex set to a new vertex attribute.

evidence$6 - (undocumented)

evidence$7 - (undocumented)

Returns:

a VertexRDD containing all the vertices in this VertexRDD with the attributes emitted by f.

innerZipJoin

public <U,
VD2> VertexRDD<VD2> innerZipJoin(VertexRDD<U> other,
 scala.Function3<Object,VD,U,VD2> f,
 scala.reflect.ClassTag<U> evidence$8,
 scala.reflect.ClassTag<VD2> evidence$9)

Description copied from class: VertexRDD

Efficiently inner joins this VertexRDD with another VertexRDD sharing the same index. See VertexRDD.<U,VD2>innerJoin(org.apache.spark.rdd.RDD<scala.Tuple2<java.lang.Object,U>>,scala.Function3<java.lang.Object,VD,U,VD2>,scala.reflect.ClassTag<U>,scala.reflect.ClassTag<VD2>) for the behavior of the join.

Specified by:

innerZipJoin in class VertexRDD<VD>

Parameters:

other - (undocumented)

f - (undocumented)

evidence$8 - (undocumented)

evidence$9 - (undocumented)

Returns:

(undocumented)

innerJoin

public <U,
VD2> VertexRDD<VD2> innerJoin(RDD<scala.Tuple2<Object,U>> other,
 scala.Function3<Object,VD,U,VD2> f,
 scala.reflect.ClassTag<U> evidence$10,
 scala.reflect.ClassTag<VD2> evidence$11)

Description copied from class: VertexRDD

Inner joins this VertexRDD with an RDD containing vertex attribute pairs. If the other RDD is backed by a VertexRDD with the same index then the efficient VertexRDD.<U,VD2>innerZipJoin(org.apache.spark.graphx.VertexRDD<U>,scala.Function3<java.lang.Object,VD,U,VD2>,scala.reflect.ClassTag<U>,scala.reflect.ClassTag<VD2>) implementation is used.

Specified by:

innerJoin in class VertexRDD<VD>

Parameters:

other - an RDD containing vertices to join. If there are multiple entries for the same vertex, one is picked arbitrarily. Use VertexRDD.<VD2>aggregateUsingIndex(org.apache.spark.rdd.RDD<scala.Tuple2<java.lang.Object,VD2>>,scala.Function2<VD2,VD2,VD2>,scala.reflect.ClassTag<VD2>) to merge multiple entries.

f - the join function applied to corresponding values of this and other

evidence$10 - (undocumented)

evidence$11 - (undocumented)

Returns:

a VertexRDD co-indexed with this, containing only vertices that appear in both this and other, with values supplied by f

aggregateUsingIndex

public <VD2> VertexRDD<VD2> aggregateUsingIndex(RDD<scala.Tuple2<Object,VD2>> messages,
 scala.Function2<VD2,VD2,VD2> reduceFunc,
 scala.reflect.ClassTag<VD2> evidence$12)

Description copied from class: VertexRDD

Aggregates vertices in messages that have the same ids using reduceFunc, returning a VertexRDD co-indexed with this.

Specified by:

aggregateUsingIndex in class VertexRDD<VD>

Parameters:

messages - an RDD containing messages to aggregate, where each message is a pair of its target vertex ID and the message data

reduceFunc - the associative aggregation function for merging messages to the same vertex

evidence$12 - (undocumented)

Returns:

a VertexRDD co-indexed with this, containing only vertices that received messages. For those vertices, their values are the result of applying reduceFunc to all received messages.

reverseRoutingTables

public VertexRDD<VD> reverseRoutingTables()

Description copied from class: VertexRDD

Returns a new VertexRDD reflecting a reversal of all edge directions in the corresponding EdgeRDD.

Specified by:

reverseRoutingTables in class VertexRDD<VD>

Returns:

(undocumented)

withEdges

public VertexRDD<VD> withEdges(EdgeRDD<?> edges)

Description copied from class: VertexRDD

Prepares this VertexRDD for efficient joins with the given EdgeRDD.

Specified by:

withEdges in class VertexRDD<VD>