| dropDuplicates {SparkR} | R Documentation |
Returns a new SparkDataFrame with duplicate rows removed, considering only the subset of columns.
dropDuplicates(x, colNames = columns(x)) ## S4 method for signature 'SparkDataFrame' dropDuplicates(x, colNames = columns(x))
x |
A SparkDataFrame. |
colnames |
A character vector of column names. |
A SparkDataFrame with duplicate rows removed.
Other SparkDataFrame functions: SparkDataFrame-class,
[[, agg,
arrange, as.data.frame,
attach, cache,
collect, colnames,
coltypes, columns,
count, dapply,
describe, dim,
distinct, dropna,
drop, dtypes,
except, explain,
filter, first,
group_by, head,
histogram, insertInto,
intersect, isLocal,
join, limit,
merge, mutate,
ncol, persist,
printSchema,
registerTempTable, rename,
repartition, sample,
saveAsTable, selectExpr,
select, showDF,
show, str,
take, unionAll,
unpersist, withColumn,
write.df, write.jdbc,
write.json, write.parquet,
write.text
## Not run:
##D sc <- sparkR.init()
##D sqlContext <- sparkRSQL.init(sc)
##D path <- "path/to/file.json"
##D df <- read.json(sqlContext, path)
##D dropDuplicates(df)
##D dropDuplicates(df, c("col1", "col2"))
## End(Not run)