Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Core Spark functionality.

  Core Spark functionality. org.apache.spark.SparkContext serves as the main entry point to Spark, while org.apache.spark.rdd.RDD is the data type representing a distributed collection, and provides most parallel operations.

  In addition, org.apache.spark.rdd.PairRDDFunctions contains operations available only on RDDs of key-value pairs, such as groupByKey and join; org.apache.spark.rdd.DoubleRDDFunctions contains operations available only on RDDs of Doubles; and org.apache.spark.rdd.SequenceFileRDDFunctions contains operations available on RDDs that can be saved as SequenceFiles. These operations are automatically available on any RDD of the right type (e.g. RDD[(Int, Int)] through implicit conversions.

  Java programmers should reference the org.apache.spark.api.java package for Spark programming APIs in Java.

  Classes and methods marked with Experimental are user-facing features which have not been officially adopted by the Spark project. These are subject to change or removal in minor releases.

  Classes and methods marked with Developer API are intended for advanced users want to extend Spark through lower level interfaces. These are subject to changes or removal in minor releases.

  Definition Classes

  apache

•  package sql

  Allows the execution of relational queries, including those expressed in SQL using Spark.

  Allows the execution of relational queries, including those expressed in SQL using Spark.

  Definition Classes

  spark

•  object functions

  Commonly used functions available for DataFrame operations.

  Commonly used functions available for DataFrame operations. Using functions defined here provides a little bit more compile-time safety to make sure the function exists.

  You can call the functions defined here by two ways: _FUNC_(...) and functions.expr("_FUNC_(...)").

  As an example, regr_count is a function that is defined here. You can use regr_count(col("yCol", col("xCol"))) to invoke the regr_count function. This way the programming language's compiler ensures regr_count exists and is of the proper form. You can also use expr("regr_count(yCol, xCol)") function to invoke the same function. In this case, Spark itself will ensure regr_count exists when it analyzes the query.

  You can find the entire list of functions at SQL API documentation of your Spark version, see also the latest list

  This function APIs usually have methods with Column signature only because it can support not only Column but also other types such as a native string. The other variants currently exist for historical reasons.

  Definition Classes

  sql

  Annotations

  @Stable()

  Since

  1.3.0

• partitioning

org.apache.spark.sql.functions

partitioning