SparkSession
Related Docs:
object SparkSession
| package sql
Apply a schema defined by the schema to an RDD.
Apply a schema defined by the schema to an RDD. It is only used by PySpark.
Apply a schema defined by the schemaString to an RDD.
Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.
Convert a BaseRelation created for external data sources into a DataFrame.
Convert a BaseRelation created for external data sources into a DataFrame.
2.0.0
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
2.0.0
Runtime configuration interface for Spark.
Runtime configuration interface for Spark.
This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.
2.0.0
Creates a DataFrame from an RDD[Row].
Creates a DataFrame from an RDD[Row]. User can specify whether the input rows should be converted to Catalyst rows.
Applies a schema to an List of Java Beans.
Applies a schema to an List of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
1.6.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
2.0.0
:: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:
import org.apache.spark.sql._ import org.apache.spark.sql.types._ val sparkSession = new org.apache.spark.sql.SparkSession(sc) val schema = StructType( StructField("name", StringType, false) :: StructField("age", IntegerType, true) :: Nil) val people = sc.textFile("examples/src/main/resources/people.txt").map( _.split(",")).map(p => Row(p(0), p(1).trim.toInt)) val dataFrame = sparkSession.createDataFrame(people, schema) dataFrame.printSchema // root // |-- name: string (nullable = false) // |-- age: integer (nullable = true) dataFrame.createOrReplaceTempView("people") sparkSession.sql("select name from people").collect.foreach(println)
2.0.0
:: Experimental :: Creates a DataFrame from a local Seq of Product.
:: Experimental :: Creates a DataFrame from a local Seq of Product.
2.0.0
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g.
:: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).
2.0.0
Creates a temporary view with a DataFrame.
Creates a temporary view with a DataFrame. The lifetime of this temporary view is tied to this SparkSession.
:: Experimental :: Returns a DataFrame with no rows or columns.
:: Experimental :: Returns a DataFrame with no rows or columns.
2.0.0
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
2.0.0
:: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
:: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
val sparkSession = SparkSession.builder.getOrCreate() import sparkSession.implicits._
2.0.0
Creates a DataFrame from an RDD[Row].
Creates a DataFrame from an RDD[Row]. User can specify whether the input rows should be converted to Catalyst rows.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
:: Experimental :: An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
2.0.0
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
Note: Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.
2.0.0
Parses the data type in our internal string representation.
Parses the data type in our internal string representation. The data type string should
have the same format as the one generated by toString in scala.
It is only used by PySpark.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with an step value, with partition number
specified.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with an step value, with partition number
specified.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with an step value.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with an step value.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from start to end (exclusive) with step value 1.
2.0.0
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from 0 to end (exclusive) with step value 1.
:: Experimental ::
Creates a Dataset with a single LongType column named id, containing elements
in an range from 0 to end (exclusive) with step value 1.
2.0.0
:: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.
:: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.
sparkSession.read.parquet("/path/to/file.parquet") sparkSession.read.schema(schema).json("/path/to/file.json")
2.0.0
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf.
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
Executes a SQL query using Spark, returning the result as a DataFrame.
Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.
2.0.0
Stop the underlying SparkContext.
Stop the underlying SparkContext.
2.0.0
Returns a ContinuousQueryManager that allows managing all the
ContinuousQueries active on this.
Returns a ContinuousQueryManager that allows managing all the
ContinuousQueries active on this.
2.0.0
Returns the specified table as a DataFrame.
Returns the specified table as a DataFrame.
2.0.0
A collection of methods for registering user-defined functions (UDF).
A collection of methods for registering user-defined functions (UDF).
The following example registers a Scala closure as UDF:
sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
The following example registers a UDF in Java:
sparkSession.udf().register("myUDF", new UDF2<Integer, String, String>() { @Override public String call(Integer arg1, String arg2) { return arg2 + arg1; } }, DataTypes.StringType);
Or, to use Java 8 lambda syntax:
sparkSession.udf().register("myUDF", (Integer arg1, String arg2) -> arg2 + arg1, DataTypes.StringType);
2.0.0
The entry point to programming Spark with the Dataset and DataFrame API.
To create a SparkSession, use the following builder pattern: