Run a function over a list of elements, distributing the computations with Spark
spark.lapply.Rd
Run a function over a list of elements, distributing the computations with Spark. Applies a function in a manner that is similar to doParallel or lapply to elements of a list. The computations are distributed using Spark. It is conceptually the same as the following code: lapply(list, func)
Details
Known limitations:
variable scoping and capture: compared to R's rich support for variable resolutions, the distributed nature of SparkR limits how variables are resolved at runtime. All the variables that are available through lexical scoping are embedded in the closure of the function and available as read-only variables within the function. The environment variables should be stored into temporary variables outside the function, and not directly accessed within the function.
loading external packages: In order to use a package, you need to load it inside the closure. For example, if you rely on the MASS module, here is how you would use it:
Examples
if (FALSE) {
sparkR.session()
doubled <- spark.lapply(1:10, function(x) {2 * x})
}