SQLContext

Companion object SQLContext

class SQLContext extends Logging with Serializable

The entry point for working with structured data (rows and columns) in Spark 1.x.

As of Spark 2.0, this is replaced by SparkSession. However, we are keeping the class here for backward compatibility.

Self Type: SQLContext
Annotations: @Stable()
Source: SQLContext.scala
Since: 1.0.0

Linear Supertypes

Serializable, Logging, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

SQLContext
Serializable
Logging
AnyRef
Any

Hide All
Show All

Visibility

Public
Protected

Instance Constructors

new SQLContext(sparkContext: JavaSparkContext)
Annotations
@deprecated
Deprecated
(Since version 2.0.0) Use SparkSession.builder instead
new SQLContext(sc: SparkContext)
Annotations
@deprecated
Deprecated
(Since version 2.0.0) Use SparkSession.builder instead

Type Members

implicit class LogStringContext extends AnyRef
Definition Classes
Logging

Value Members

final def !=(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def ##: Int
Definition Classes
AnyRef → Any
final def ==(arg0: Any): Boolean
Definition Classes
AnyRef → Any
final def asInstanceOf[T0]: T0
Definition Classes
Any
def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame
Convert a BaseRelation created for external data sources into a DataFrame.
Convert a BaseRelation created for external data sources into a DataFrame.
Since
1.3.0
def cacheTable(tableName: String): Unit
Caches the specified table in-memory.
Caches the specified table in-memory.
Since
1.3.0
def clearCache(): Unit
Removes all cached tables from the in-memory cache.
Removes all cached tables from the in-memory cache.
Since
1.3.0
def clone(): AnyRef
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.CloneNotSupportedException]) @IntrinsicCandidate() @native()
def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame
Applies a schema to a List of Java Beans.
Applies a schema to a List of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
Since
1.6.0
def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
Since
1.3.0
def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame
Applies a schema to an RDD of Java Beans.
Applies a schema to an RDD of Java Beans.
WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.
Since
1.3.0
def createDataFrame(rows: List[Row], schema: StructType): DataFrame
:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from a java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.
Annotations
@DeveloperApi()
Since
1.6.0
def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame
:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema.
:: DeveloperApi :: Creates a DataFrame from a JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.
Annotations
@DeveloperApi()
Since
1.3.0

def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

:: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

import org.apache.spark.sql._
import org.apache.spark.sql.types._
val sqlContext = new org.apache.spark.sql.SQLContext(sc)

val schema =
  StructType(
    StructField("name", StringType, false) ::
    StructField("age", IntegerType, true) :: Nil)

val people =
  sc.textFile("examples/src/main/resources/people.txt").map(
    _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
val dataFrame = sqlContext.createDataFrame(people, schema)
dataFrame.printSchema
// root
// |-- name: string (nullable = false)
// |-- age: integer (nullable = true)

dataFrame.createOrReplaceTempView("people")
sqlContext.sql("select name from people").collect.foreach(println)

Annotations: @DeveloperApi()
Since: 1.3.0

def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame
Creates a DataFrame from a local Seq of Product.
Creates a DataFrame from a local Seq of Product.
Since
1.3.0
def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame
Creates a DataFrame from an RDD of Product (e.g.
Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).
Since
1.3.0
def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]
Creates a Dataset from a java.util.List of a given type.
Creates a Dataset from a java.util.List of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.
Java Example
```
List<String> data = Arrays.asList("hello", "world");
Dataset<String> ds = spark.createDataset(data, Encoders.STRING());
```
Since
2.0.0
def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]
Creates a Dataset from an RDD of a given type.
Creates a Dataset from an RDD of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.
Since
2.0.0
def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]
Creates a Dataset from a local Seq of data of a given type.
Creates a Dataset from a local Seq of data of a given type. This method requires an encoder (to convert a JVM object of type T to and from the internal Spark SQL representation) that is generally created automatically through implicits from a SparkSession, or can be created explicitly by calling static methods on Encoders.
Example
```
import spark.implicits._
case class Person(name: String, age: Long)
val data = Seq(Person("Michael", 29), Person("Andy", 30), Person("Justin", 19))
val ds = spark.createDataset(data)

ds.show()
// +-------+---+
// |   name|age|
// +-------+---+
// |Michael| 29|
// |   Andy| 30|
// | Justin| 19|
// +-------+---+
```
Since
2.0.0
def dropTempTable(tableName: String): Unit
Drops the temporary table with the given table name in the catalog.
Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.
tableName
the name of the table to be unregistered.
Since
1.3.0
def emptyDataFrame: DataFrame
Returns a DataFrame with no rows or columns.
Returns a DataFrame with no rows or columns.
Since
1.3.0
final def eq(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def equals(arg0: AnyRef): Boolean
Definition Classes
AnyRef → Any
def experimental: ExperimentalMethods
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
:: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
Annotations
@Experimental() @transient() @Unstable()
Since
1.3.0
def getAllConfs: Map[String, String]
Return all the configuration properties that have been set (i.e.
Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.
Since
1.0.0
final def getClass(): Class[_ <: AnyRef]
Definition Classes
AnyRef → Any
Annotations
@IntrinsicCandidate() @native()
def getConf(key: String, defaultValue: String): String
Return the value of Spark SQL configuration property for the given key.
Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.
Since
1.0.0
def getConf(key: String): String
Return the value of Spark SQL configuration property for the given key.
Return the value of Spark SQL configuration property for the given key.
Since
1.0.0
def hashCode(): Int
Definition Classes
AnyRef → Any
Annotations
@IntrinsicCandidate() @native()
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit
Attributes
protected
Definition Classes
Logging
def isCached(tableName: String): Boolean
Returns true if the table is currently cached in-memory.
Returns true if the table is currently cached in-memory.
Since
1.3.0
final def isInstanceOf[T0]: Boolean
Definition Classes
Any
def isTraceEnabled(): Boolean
Attributes
protected
Definition Classes
Logging
def listenerManager: ExecutionListenerManager
An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.
def log: Logger
Attributes
protected
Definition Classes
Logging
def logDebug(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logDebug(entry: LogEntry, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logDebug(entry: LogEntry): Unit
Attributes
protected
Definition Classes
Logging
def logDebug(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logError(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logError(entry: LogEntry, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logError(entry: LogEntry): Unit
Attributes
protected
Definition Classes
Logging
def logError(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(entry: LogEntry, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(entry: LogEntry): Unit
Attributes
protected
Definition Classes
Logging
def logInfo(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logName: String
Attributes
protected
Definition Classes
Logging
def logTrace(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logTrace(entry: LogEntry, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logTrace(entry: LogEntry): Unit
Attributes
protected
Definition Classes
Logging
def logTrace(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(msg: => String, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(entry: LogEntry, throwable: Throwable): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(entry: LogEntry): Unit
Attributes
protected
Definition Classes
Logging
def logWarning(msg: => String): Unit
Attributes
protected
Definition Classes
Logging
final def ne(arg0: AnyRef): Boolean
Definition Classes
AnyRef
def newSession(): SQLContext
Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, cached data and other things.
Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, cached data and other things.
Since
1.6.0
final def notify(): Unit
Definition Classes
AnyRef
Annotations
@IntrinsicCandidate() @native()
final def notifyAll(): Unit
Definition Classes
AnyRef
Annotations
@IntrinsicCandidate() @native()
def range(start: Long, end: Long, step: Long, numPartitions: Int): DataFrame
Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.
Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.
Since
1.4.0
def range(start: Long, end: Long, step: Long): DataFrame
Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.
Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.
Since
2.0.0
def range(start: Long, end: Long): DataFrame
Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.
Creates a DataFrame with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.
Since
1.4.0
def range(end: Long): DataFrame
Creates a DataFrame with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.
Creates a DataFrame with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.
Since
1.4.1
def read: DataFrameReader
Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
```
sqlContext.read.parquet("/path/to/file.parquet")
sqlContext.read.schema(schema).json("/path/to/file.json")
```
Since
1.4.0
def readStream: DataStreamReader
Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
```
sparkSession.readStream.parquet("/path/to/directory/of/parquet/files")
sparkSession.readStream.schema(schema).json("/path/to/directory/of/json/files")
```
Since
2.0.0
def setConf(key: String, value: String): Unit
Set the given Spark SQL configuration property.
Set the given Spark SQL configuration property.
Since
1.0.0
def setConf(props: Properties): Unit
Set Spark SQL configuration properties.
Set Spark SQL configuration properties.
Since
1.0.0
def sparkContext: SparkContext
val sparkSession: SparkSession
def sql(sqlText: String): DataFrame
Executes a SQL query using Spark, returning the result as a DataFrame.
Executes a SQL query using Spark, returning the result as a DataFrame. This API eagerly runs DDL/DML commands, but not for SELECT queries.
Since
1.3.0
def streams: StreamingQueryManager
Returns a StreamingQueryManager that allows managing all the StreamingQueries active on this context.
Returns a StreamingQueryManager that allows managing all the StreamingQueries active on this context.
Since
2.0.0
final def synchronized[T0](arg0: => T0): T0
Definition Classes
AnyRef
def table(tableName: String): DataFrame
Returns the specified table as a DataFrame.
Returns the specified table as a DataFrame.
Since
1.3.0
def tableNames(databaseName: String): Array[String]
Returns the names of tables in the given database as an array.
Returns the names of tables in the given database as an array.
Since
1.3.0
def tableNames(): Array[String]
Returns the names of tables in the current database as an array.
Returns the names of tables in the current database as an array.
Since
1.3.0
def tables(databaseName: String): DataFrame
Returns a DataFrame containing names of existing tables in the given database.
Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has three columns, database, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).
Since
1.3.0
def tables(): DataFrame
Returns a DataFrame containing names of existing tables in the current database.
Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has three columns, database, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).
Since
1.3.0
def toString(): String
Definition Classes
AnyRef → Any
def udf: UDFRegistration
A collection of methods for registering user-defined functions (UDF).
A collection of methods for registering user-defined functions (UDF).
The following example registers a Scala closure as UDF:
```
sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)
```
The following example registers a UDF in Java:
```
sqlContext.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);
```
Since
1.3.0
Note
The user-defined functions must be deterministic. Due to optimization, duplicate invocations may be eliminated or the function may even be invoked more times than it is present in the query.
def uncacheTable(tableName: String): Unit
Removes the specified table from the in-memory cache.
Removes the specified table from the in-memory cache.
Since
1.3.0
final def wait(arg0: Long, arg1: Int): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
final def wait(arg0: Long): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException]) @native()
final def wait(): Unit
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.InterruptedException])
def withLogContext(context: HashMap[String, String])(body: => Unit): Unit
Attributes
protected
Definition Classes
Logging
object implicits extends SQLImplicits with Serializable
(Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
(Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
```
val sqlContext = new SQLContext(sc)
import sqlContext.implicits._
```
Since
1.3.0

Deprecated Value Members

def applySchema(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame
Annotations
@deprecated
Deprecated
(Since version 1.3.0) Use createDataFrame instead.
def applySchema(rdd: RDD[_], beanClass: Class[_]): DataFrame
Annotations
@deprecated
Deprecated
(Since version 1.3.0) Use createDataFrame instead.
def applySchema(rowRDD: JavaRDD[Row], schema: StructType): DataFrame
Annotations
@deprecated
Deprecated
(Since version 1.3.0) Use createDataFrame instead.
def applySchema(rowRDD: RDD[Row], schema: StructType): DataFrame
Annotations
@deprecated
Deprecated
(Since version 1.3.0) Use createDataFrame instead.
def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
(Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.
(Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame
Create an external table from the given path based on a data source, a schema and a set of options.
Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame
(Scala-specific) Creates an external table from the given path based on a data source and a set of options.
(Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame
Creates an external table from the given path based on a data source and a set of options.
Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def createExternalTable(tableName: String, path: String, source: String): DataFrame
Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
Creates an external table from the given path based on a data source and returns the corresponding DataFrame.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def createExternalTable(tableName: String, path: String): DataFrame
Creates an external table from the given path and returns the corresponding DataFrame.
Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.
Annotations
@deprecated
Deprecated
(Since version 2.2.0) use sparkSession.catalog.createTable instead.
Since
1.3.0
def finalize(): Unit
Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws(classOf[java.lang.Throwable]) @Deprecated
Deprecated
(Since version 9)
def jdbc(url: String, table: String, theParts: Array[String]): DataFrame
Construct a DataFrame representing the database table accessible via JDBC URL url named table.
Construct a DataFrame representing the database table accessible via JDBC URL url named table. The theParts parameter gives a list expressions suitable for inclusion in WHERE clauses; each one defines one partition of the DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.jdbc() instead.
def jdbc(url: String, table: String, columnName: String, lowerBound: Long, upperBound: Long, numPartitions: Int): DataFrame
Construct a DataFrame representing the database table accessible via JDBC URL url named table.
Construct a DataFrame representing the database table accessible via JDBC URL url named table. Partitions of the table will be retrieved in parallel based on the parameters passed to this function.
columnName
the name of a column of integral type that will be used for partitioning.
lowerBound
the minimum value of columnName used to decide partition stride
upperBound
the maximum value of columnName used to decide partition stride
numPartitions
the number of partitions. the range minValue-maxValue will be split evenly into this many partitions
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.jdbc() instead.
def jdbc(url: String, table: String): DataFrame
Construct a DataFrame representing the database table accessible via JDBC URL url named table.
Construct a DataFrame representing the database table accessible via JDBC URL url named table.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.jdbc() instead.
def jsonFile(path: String, samplingRatio: Double): DataFrame
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonFile(path: String, schema: StructType): DataFrame
Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.
Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonFile(path: String): DataFrame
Loads a JSON file (one object per line), returning the result as a DataFrame.
Loads a JSON file (one object per line), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: JavaRDD[String], samplingRatio: Double): DataFrame
Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.
Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: RDD[String], samplingRatio: Double): DataFrame
Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.
Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: JavaRDD[String], schema: StructType): DataFrame
Loads an JavaRDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.
Loads an JavaRDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: RDD[String], schema: StructType): DataFrame
Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.
Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: JavaRDD[String]): DataFrame
Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.
Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def jsonRDD(json: RDD[String]): DataFrame
Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.
Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.json() instead.
def load(source: String, schema: StructType, options: Map[String, String]): DataFrame
(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.
(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.format(source).schema(schema).options(options).load() instead.
def load(source: String, schema: StructType, options: Map[String, String]): DataFrame
(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.
(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.format(source).schema(schema).options(options).load() instead.
def load(source: String, options: Map[String, String]): DataFrame
(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.
(Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.format(source).options(options).load() instead.
def load(source: String, options: Map[String, String]): DataFrame
(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.
(Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.format(source).options(options).load() instead.
def load(path: String, source: String): DataFrame
Returns the dataset stored at path as a DataFrame, using the given data source.
Returns the dataset stored at path as a DataFrame, using the given data source.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.format(source).load(path) instead.
def load(path: String): DataFrame
Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.
Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.
Annotations
@deprecated
Deprecated
(Since version 1.4.0) Use read.load(path) instead.
def parquetFile(paths: String*): DataFrame
Loads a Parquet file, returning the result as a DataFrame.
Loads a Parquet file, returning the result as a DataFrame. This function returns an empty DataFrame if no paths are passed in.
Annotations
@deprecated @varargs()
Deprecated
(Since version 1.4.0) Use read.parquet() instead.

Packages

SQLContext

Companion object SQLContext

class SQLContext extends Logging with Serializable

Instance Constructors

Type Members

Value Members

Java Example

Example

Deprecated Value Members

Inherited from Serializable

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Basic Operations

Cached Table Management

Configuration

dataframe

Custom DataFrame Creation

Custom Dataset Creation

Persistent Catalog DDL

Generic Data Sources

Specific Data Sources

Support functions for language integrated queries

Packages

SQLContext

Companion object SQLContext

class SQLContext extends Logging with Serializable

Instance Constructors

Type Members

Value Members

Java Example

Example

Deprecated Value Members

Inherited from Serializable

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Basic Operations

Cached Table Management

Configuration

dataframe

Custom DataFrame Creation

Custom Dataset Creation

Persistent Catalog DDL

Generic Data Sources

Specific Data Sources

Support functions for language integrated queries

SQLContext