public class SQLContext
extends Object
implements com.typesafe.scalalogging.slf4j.Logging, scala.Serializable
SchemaRDD
objects and the execution of SQL queries.
Constructor and Description |
---|
SQLContext(SparkContext sparkContext) |
Modifier and Type | Method and Description |
---|---|
void |
cacheTable(String tableName)
Caches the specified table in-memory.
|
<A extends scala.Product> |
createParquetFile(String path,
boolean allowExisting,
org.apache.hadoop.conf.Configuration conf,
scala.reflect.api.TypeTags.TypeTag<A> evidence$2)
:: Experimental ::
Creates an empty parquet file with the schema of class
A , which can be registered as a table. |
<A extends scala.Product> |
createSchemaRDD(RDD<A> rdd,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
Creates a SchemaRDD from an RDD of case classes.
|
SchemaRDD |
logicalPlanToSparkQuery(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
:: DeveloperApi ::
Allows catalyst LogicalPlans to be executed as a SchemaRDD.
|
SchemaRDD |
parquetFile(String path)
Loads a Parquet file, returning the result as a
SchemaRDD . |
void |
registerRDDAsTable(SchemaRDD rdd,
String tableName)
Registers the given RDD as a temporary table in the catalog.
|
SparkContext |
sparkContext() |
SchemaRDD |
sql(String sqlText)
Executes a SQL query using Spark, returning the result as a SchemaRDD.
|
SchemaRDD |
table(String tableName)
Returns the specified table as a SchemaRDD
|
void |
uncacheTable(String tableName)
Removes the specified table from the in-memory cache.
|
public SQLContext(SparkContext sparkContext)
public SparkContext sparkContext()
public SchemaRDD logicalPlanToSparkQuery(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
public <A extends scala.Product> SchemaRDD createSchemaRDD(RDD<A> rdd, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
public SchemaRDD parquetFile(String path)
SchemaRDD
.
public <A extends scala.Product> SchemaRDD createParquetFile(String path, boolean allowExisting, org.apache.hadoop.conf.Configuration conf, scala.reflect.api.TypeTags.TypeTag<A> evidence$2)
A
, which can be registered as a table.
This registered table can be used as the target of future insertInto
operations.
val sqlContext = new SQLContext(...)
import sqlContext._
case class Person(name: String, age: Int)
createParquetFile[Person]("path/to/file.parquet").registerAsTable("people")
sql("INSERT INTO people SELECT 'michael', 29")
path
- The path where the directory containing parquet metadata should be created.
Data inserted into this table will also be stored at this location.allowExisting
- When false, an exception will be thrown if this directory already exists.conf
- A Hadoop configuration object that can be used to specify options to the parquet
output format.
public void registerRDDAsTable(SchemaRDD rdd, String tableName)
public SchemaRDD sql(String sqlText)
public SchemaRDD table(String tableName)
public void cacheTable(String tableName)
public void uncacheTable(String tableName)