Add a single artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

Supported target file extensions are .jar and .class.

Example

addArtifact("/Users/dummyUser/files/foo/bar.class", "foo/bar.class")
addArtifact("/Users/dummyUser/files/flat.class", "flat.class")
// Directory structure of the session's working directory for class files would look like:
// ${WORKING_DIR_FOR_CLASS_FILES}/flat.class
// ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class

Add a single in-memory artifact to the session while preserving the directory structure specified by target under the session's working directory of that particular file extension.

Supported target file extensions are .jar and .class.

Example

addArtifact(bytesBar, "foo/bar.class")
addArtifact(bytesFlat, "flat.class")
// Directory structure of the session's working directory for class files would look like:
// ${WORKING_DIR_FOR_CLASS_FILES}/flat.class
// ${WORKING_DIR_FOR_CLASS_FILES}/foo/bar.class

Add a single artifact to the current session.

Currently it supports local files with extensions .jar and .class and Apache Ivy URIs.

Add a single artifact to the current session.

Currently only local files with extensions .jar and .class are supported.

Add one or more artifacts to the session.

Currently it supports local files with extensions .jar and .class and Apache Ivy URIs

Convert a BaseRelation created for external data sources into a DataFrame.

Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.

Stop the underlying SparkContext.

Runtime configuration interface for Spark.

This is the interface through which the user can get and set all Spark and Hadoop configurations that are relevant to Spark SQL. When getting the value of a config, this defaults to the value set in the underlying SparkContext, if any.

Applies a schema to a List of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

Applies a schema to an RDD of Java Beans.

WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

:: DeveloperApi :: Creates a DataFrame from a java.util.List containing org.apache.spark.sql.Rows using the given schema.It is important to make sure that the structure of every org.apache.spark.sql.Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

import org.apache.spark.sql._
import org.apache.spark.sql.types._
val sparkSession = new org.apache.spark.sql.SparkSession(sc)

val schema =
  StructType(
    StructField("name", StringType, false) ::
    StructField("age", IntegerType, true) :: Nil)

val people =
  sc.textFile("examples/src/main/resources/people.txt").map(
    _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
val dataFrame = sparkSession.createDataFrame(people, schema)
dataFrame.printSchema
// root
// |-- name: string (nullable = false)
// |-- age: integer (nullable = true)

dataFrame.createOrReplaceTempView("people")
sparkSession.sql("select name from people").collect.foreach(println)

Creates a DataFrame from a local Seq of Product.

Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Java Example

List<String> data = Arrays.asList("hello", "world");
Dataset<String> ds = spark.createDataset(data, Encoders.STRING());

Creates a Dataset from an RDD of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.

Example

import spark.implicits._
case class Person(name: String, age: Long)
val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
val ds = spark.createDataset(data)

ds.show()
// +-------+---+
// |   name|age|
// +-------+---+
// |Michael| 29|
// |   Andy| 30|
// | Justin| 19|
// +-------+---+

A collection of methods for registering user-defined data sources.

Returns a DataFrame with no rows or columns.

Creates a new Dataset of type T containing zero elements.

Execute an arbitrary string command inside an external execution engine rather than Spark. This could be useful when user wants to execute some commands out of Spark. For example, executing custom DDL/DML command for JDBC, creating index for ElasticSearch, creating cores for Solr and so on.

The command will be eagerly executed after this method is called and the returned DataFrame will contain the output of the command(if any).

:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.

Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.

Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.

Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.

Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.

sparkSession.read.parquet("/path/to/file.parquet")
sparkSession.read.schema(schema).json("/path/to/file.json")

Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.

sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")

State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a org.apache.spark.sql.internal.SQLConf. If parentSessionState is not null, the SessionState will be a copy of the parent.

This is internal to Spark and there is no guarantee on interface stability.

State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.

This is internal to Spark and there is no guarantee on interface stability.

Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting named parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

Executes a SQL query substituting positional parameters by the given arguments, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.

A wrapped version of this session in the form of a SQLContext, for backward compatibility.

Synonym for close().

Returns a StreamingQueryManager that allows managing all the StreamingQuerys active on this.

Returns the specified table/view as a DataFrame. If it's a table, it must support batch reading and the returned DataFrame is the batch scan query plan of this table. If it's a view, the returned DataFrame is simply the query plan of the view, which can either be a batch or streaming query plan.

Executes some code block and prints to stdout the time taken to execute the block. This is available in Scala only and is used primarily for interactive testing and debugging.

A collection of methods for registering user-defined functions (UDF).

The following example registers a Scala closure as UDF:

sparkSession.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

The following example registers a UDF in Java:

sparkSession.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);

The version of Spark on which this application is running.

(Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

val sparkSession = SparkSession.builder.getOrCreate()
import sparkSession.implicits._