org.apache.spark.sql.api.SparkSession<DS>

All Implemented Interfaces:: Closeable, Serializable, AutoCloseable

Direct Known Subclasses:: SparkSession

public abstract class SparkSession<DS extends Dataset<Object,DS>> extends Object implements Serializable, Closeable

The entry point to programming Spark with the Dataset and DataFrame API.

In environments that this has been created upfront (e.g. REPL, notebooks), use the builder to get an existing session:


   SparkSession.builder().getOrCreate()

The builder can also be used to create a new session:


   SparkSession.builder
     .master("local")
     .appName("Word Count")
     .config("spark.some.config.option", "some-value")
     .getOrCreate()

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor

Description

SparkSession()
Method Summary

Modifier and Type

Method

Description

abstract void

addArtifact(byte[] bytes, String target)

Add a single in-memory artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

abstract void

addArtifact(String path)

Add a single artifact to the current session.

abstract void

addArtifact(String source, String target)

Add a single artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

abstract void

addArtifact(URI uri)

Add a single artifact to the current session.

void

addArtifacts(URI... uri)

Add one or more artifacts to the session.

abstract void

addArtifacts(scala.collection.immutable.Seq<URI> uri)

Add one or more artifacts to the session.

abstract Catalog<DS>

catalog()

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

abstract DS

createDataFrame(List<?> data, Class<?> beanClass)

Applies a schema to a List of Java Beans.

abstract DS

createDataFrame(List<Row> rows, StructType schema)

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.It is important to make sure that the structure of every Row of the provided List matches the provided schema.

abstract <A extends scala.Product> DS

createDataFrame(scala.collection.immutable.Seq<A> data, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)

Creates a DataFrame from a local Seq of Product.

abstract <T> DS

createDataset(List<T> data, Encoder<T> evidence$4)

Creates a Dataset from a java.util.List of a given type.

abstract <T> DS

createDataset(scala.collection.immutable.Seq<T> data, Encoder<T> evidence$3)

Creates a Dataset from a local Seq of data of a given type.

abstract DS

emptyDataFrame()

Returns a DataFrame with no rows or columns.

abstract <T> DS

emptyDataset(Encoder<T> evidence$2)

Creates a new Dataset of type T containing zero elements.

abstract SparkSession<DS>

newSession()

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

abstract DS

range(long end)

Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

abstract DS

range(long start, long end)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

abstract DS

range(long start, long end, long step)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

abstract DS

range(long start, long end, long step, int numPartitions)

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

abstract DataFrameReader<DS>

read()

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

DS

sql(String sqlText)

Executes a SQL query using Spark, returning the result as a DataFrame.

abstract DS

sql(String sqlText, Object args)

Executes a SQL query substituting positional parameters by the given arguments, returning the result as a DataFrame.

DS

sql(String sqlText, Map<String,Object> args)

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

abstract DS

sql(String sqlText, scala.collection.immutable.Map<String,Object> args)

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame.

void

stop()

Synonym for close().

abstract DS

table(String tableName)

Returns the specified table/view as a DataFrame.

<T> T

time(scala.Function0<T> f)

Executes some code block and prints to stdout the time taken to execute the block.

abstract UDFRegistration

udf()

A collection of methods for registering user-defined functions (UDF).

abstract String

version()

The version of Spark on which this application is running.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.io.Closeable
close

Constructor Details
- SparkSession
  
  public SparkSession()
Method Details
- addArtifact
  
  public abstract void addArtifact(String path)
  
  Add a single artifact to the current session.
  Currently only local files with extensions .jar and .class are supported.
  
  Parameters:
  
  path - (undocumented)
  
  Since:
  
  4.0.0
- addArtifact
  
  public abstract void addArtifact(URI uri)
  
  Add a single artifact to the current session.
  Currently it supports local files with extensions .jar and .class and Apache Ivy URIs.
  
  Parameters:
  
  uri - (undocumented)
  
  Since:
  
  4.0.0
- addArtifact
  
  public abstract void addArtifact(byte[] bytes, String target)
  Add a single in-memory artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.
  Supported target file extensions are .jar and .class.
  ==Example==
  addArtifact(bytesBar, "foo/bar.class") addArtifact(bytesFlat, "flat.class") // Directory structure of the session's working directory for class files would look like: // ${WORKING_DIR_FOR_CLASS_FILES}/flat.class // ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class
  Parameters:
  
  bytes - (undocumented)
  
  target - (undocumented)
  
  Since:
  
  4.0.0
- addArtifact
  
  public abstract void addArtifact(String source, String target)
  Add a single artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.
  Supported target file extensions are .jar and .class.
  ==Example==
  addArtifact("/Users/dummyUser/files/foo/bar.class", "foo/bar.class") addArtifact("/Users/dummyUser/files/flat.class", "flat.class") // Directory structure of the session's working directory for class files would look like: // ${WORKING_DIR_FOR_CLASS_FILES}/flat.class // ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class
  Parameters:
  
  source - (undocumented)
  
  target - (undocumented)
  
  Since:
  
  4.0.0
- addArtifacts
  
  public void addArtifacts(URI... uri)
  
  Add one or more artifacts to the session.
  Currently it supports local files with extensions .jar and .class and Apache Ivy URIs
  
  Parameters:
  
  uri - (undocumented)
  
  Since:
  
  4.0.0
- addArtifacts
  
  public abstract void addArtifacts(scala.collection.immutable.Seq<URI> uri)
  
  Add one or more artifacts to the session.
  Currently it supports local files with extensions .jar and .class and Apache Ivy URIs
  
  Parameters:
  
  uri - (undocumented)
  
  Since:
  
  4.0.0
- catalog
  
  public abstract Catalog<DS> catalog()
  
  Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- createDataFrame
  
  public abstract <A extends scala.Product> DS createDataFrame(scala.collection.immutable.Seq<A> data, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
  
  Creates a DataFrame from a local Seq of Product.
  
  Parameters:
  
  data - (undocumented)
  
  evidence$1 - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- createDataFrame
  
  public abstract DS createDataFrame(List<Row> rows, StructType schema)
  
  :: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.
  
  Parameters:
  
  rows - (undocumented)
  
  schema - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- createDataFrame
  
  public abstract DS createDataFrame(List<?> data, Class<?> beanClass)
  
  Applies a schema to a List of Java Beans.
  WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
  
  Parameters:
  
  data - (undocumented)
  
  beanClass - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  1.6.0
- createDataset
  
  public abstract <T> DS createDataset(scala.collection.immutable.Seq<T> data, Encoder<T> evidence$3)
  Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.
  ==Example==
  
  import spark.implicits._ case class Person(name: String, age: Long) val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19)) val ds = spark.createDataset(data) ds.show() // +-------+---+ // | name|age| // +-------+---+ // |Michael| 29| // | Andy| 30| // | Justin| 19| // +-------+---+
  Parameters:
  
  data - (undocumented)
  
  evidence$3 - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- createDataset
  
  public abstract <T> DS createDataset(List<T> data, Encoder<T> evidence$4)
  Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.
  ==Java Example==
  
  List<String> data = Arrays.asList("hello", "world"); Dataset<String> ds = spark.createDataset(data, Encoders.STRING());
  Parameters:
  
  data - (undocumented)
  
  evidence$4 - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- emptyDataFrame
  
  public abstract DS emptyDataFrame()
  
  Returns a DataFrame with no rows or columns.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- emptyDataset
  
  public abstract <T> DS emptyDataset(Encoder<T> evidence$2)
  
  Creates a new Dataset of type T containing zero elements.
  
  Parameters:
  
  evidence$2 - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- newSession
  
  public abstract SparkSession<DS> newSession()
  
  Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
  
  Note:
  
  Other than the SparkContext, all shared state is initialized lazily. This method will force the initialization of the shared state to ensure that parent and child sessions are set up with the same shared state. If the underlying catalog implementation is Hive, this will initialize the metastore, which may take some time.
- range
  
  public abstract DS range(long end)
  
  Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.
  
  Parameters:
  
  end - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- range
  
  public abstract DS range(long start, long end)
  
  Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.
  
  Parameters:
  
  start - (undocumented)
  
  end - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- range
  
  public abstract DS range(long start, long end, long step)
  
  Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.
  
  Parameters:
  
  start - (undocumented)
  
  end - (undocumented)
  
  step - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- range
  
  public abstract DS range(long start, long end, long step, int numPartitions)
  
  Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.
  
  Parameters:
  
  start - (undocumented)
  
  end - (undocumented)
  
  step - (undocumented)
  
  numPartitions - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- read
  
  public abstract DataFrameReader<DS> read()
  Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
  sparkSession.read.parquet("/path/to/file.parquet") sparkSession.read.schema(schema).json("/path/to/file.json")
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- sql
  
  public abstract DS sql(String sqlText, Object args)
  
  Executes a SQL query substituting positional parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.
  
  Parameters:
  
  sqlText - A SQL statement with positional parameters to execute.
  
  args - An array of Java/Scala objects that can be converted to SQL literal expressions. See Supported Data Types for supported value types in Scala/Java. For example, 1, "Steven", LocalDate.of(2023, 4, 2). A value can be also a Column of a literal or collection constructor functions such as map(), array(), struct(), in that case it is taken as is.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  3.5.0
- sql
  
  public abstract DS sql(String sqlText, scala.collection.immutable.Map<String,Object> args)
  
  Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.
  
  Parameters:
  
  sqlText - A SQL statement with named parameters to execute.
  
  args - A map of parameter names to Java/Scala objects that can be converted to SQL literal expressions. See Supported Data Types for supported value types in Scala/Java. For example, map keys: "rank", "name", "birthdate"; map values: 1, "Steven", LocalDate.of(2023, 4, 2). Map value can be also a Column of a literal or collection constructor functions such as map(), array(), struct(), in that case it is taken as is.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  3.4.0
- sql
  
  public DS sql(String sqlText, Map<String,Object> args)
  
  Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.
  
  Parameters:
  
  sqlText - A SQL statement with named parameters to execute.
  
  args - A map of parameter names to Java/Scala objects that can be converted to SQL literal expressions. See Supported Data Types for supported value types in Scala/Java. For example, map keys: "rank", "name", "birthdate"; map values: 1, "Steven", LocalDate.of(2023, 4, 2). Map value can be also a Column of a literal or collection constructor functions such as map(), array(), struct(), in that case it is taken as is.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  3.4.0
- sql
  
  public DS sql(String sqlText)
  
  Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.
  
  Parameters:
  
  sqlText - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- stop
  
  public void stop()
  
  Synonym for close().
  
  Since:
  
  2.0.0
- table
  
  public abstract DS table(String tableName)
  
  Returns the specified table/view as a DataFrame. If it's a table, it must support batch reading and the returned DataFrame is the batch scan query plan of this table. If it's a view, the returned DataFrame is simply the query plan of the view, which can either be a batch or streaming query plan.
  
  Parameters:
  
  tableName - is either a qualified or unqualified name that designates a table or view. If a database is specified, it identifies the table/view from the database. Otherwise, it first attempts to find a temporary view with the given name and then match the table/view from the current database. Note that, the global temporary view database is also valid here.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
- time
  
  public <T> T time(scala.Function0<T> f)
  
  Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.
  
  Parameters:
  
  f - (undocumented)
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.1.0
- udf
  
  public abstract UDFRegistration udf()
  A collection of methods for registering user-defined functions (UDF).
  The following example registers a Scala closure as UDF:
  sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
  
  The following example registers a UDF in Java:
  sparkSession.udf().register("myUDF", (Integer arg1, String arg2) -> arg2 + arg1, DataTypes.StringType);
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0
  
  Note:
  
  The user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.
- version
  
  public abstract String version()
  
  The version of Spark on which this application is running.
  
  Returns:
  
  (undocumented)
  
  Since:
  
  2.0.0

Class SparkSession<DS extends Dataset<Object,DS>>

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Methods inherited from interface java.io.Closeable

Constructor Details

SparkSession

Method Details

addArtifact

addArtifact

addArtifact

addArtifact

addArtifacts

addArtifacts

catalog

createDataFrame

createDataFrame

createDataFrame

createDataset

createDataset

emptyDataFrame

emptyDataset

newSession

range

range

range

range

read

sql

sql

sql

sql

stop

table

time

udf

version