pyspark.RDD.distinct

RDD.distinct(numPartitions=None)

Return a new RDD containing the distinct elements in this RDD.

Examples

>>> sorted(sc.parallelize([1, 1, 2, 3]).distinct().collect())
[1, 2, 3]

pyspark.RDD.countByValue pyspark.RDD.filter