public class SQLContext extends Object implements com.typesafe.scalalogging.slf4j.Logging, SQLConf, scala.Serializable
SchemaRDD
objects and the execution of SQL queries.
Constructor and Description |
---|
SQLContext(SparkContext sparkContext) |
Modifier and Type | Method and Description |
---|---|
void |
cacheTable(String tableName)
Caches the specified table in-memory.
|
<A extends scala.Product> |
createParquetFile(String path,
boolean allowExisting,
org.apache.hadoop.conf.Configuration conf,
scala.reflect.api.TypeTags.TypeTag<A> evidence$2)
:: Experimental ::
Creates an empty parquet file with the schema of class
A , which can be registered as a table. |
<A extends scala.Product> |
createSchemaRDD(RDD<A> rdd,
scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
Creates a SchemaRDD from an RDD of case classes.
|
boolean |
isCached(String tableName)
Returns true if the table is currently cached in-memory.
|
SchemaRDD |
jsonFile(String path)
Loads a JSON file (one object per line), returning the result as a
SchemaRDD . |
SchemaRDD |
jsonFile(String path,
double samplingRatio)
:: Experimental ::
|
SchemaRDD |
jsonRDD(RDD<String> json)
Loads an RDD[String] storing JSON objects (one object per record), returning the result as a
SchemaRDD . |
SchemaRDD |
jsonRDD(RDD<String> json,
double samplingRatio)
:: Experimental ::
|
SchemaRDD |
logicalPlanToSparkQuery(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
:: DeveloperApi ::
Allows catalyst LogicalPlans to be executed as a SchemaRDD.
|
SchemaRDD |
parquetFile(String path)
Loads a Parquet file, returning the result as a
SchemaRDD . |
void |
registerRDDAsTable(SchemaRDD rdd,
String tableName)
Registers the given RDD as a temporary table in the catalog.
|
SparkContext |
sparkContext() |
SchemaRDD |
sql(String sqlText)
Executes a SQL query using Spark, returning the result as a SchemaRDD.
|
SchemaRDD |
table(String tableName)
Returns the specified table as a SchemaRDD
|
void |
uncacheTable(String tableName)
Removes the specified table from the in-memory cache.
|
public SQLContext(SparkContext sparkContext)
public SparkContext sparkContext()
public SchemaRDD logicalPlanToSparkQuery(org.apache.spark.sql.catalyst.plans.logical.LogicalPlan plan)
public <A extends scala.Product> SchemaRDD createSchemaRDD(RDD<A> rdd, scala.reflect.api.TypeTags.TypeTag<A> evidence$1)
public SchemaRDD parquetFile(String path)
SchemaRDD
.
public SchemaRDD jsonFile(String path)
SchemaRDD
.
It goes through the entire dataset once to determine the schema.
public SchemaRDD jsonFile(String path, double samplingRatio)
public SchemaRDD jsonRDD(RDD<String> json)
SchemaRDD
.
It goes through the entire dataset once to determine the schema.
public <A extends scala.Product> SchemaRDD createParquetFile(String path, boolean allowExisting, org.apache.hadoop.conf.Configuration conf, scala.reflect.api.TypeTags.TypeTag<A> evidence$2)
A
, which can be registered as a table.
This registered table can be used as the target of future insertInto
operations.
val sqlContext = new SQLContext(...)
import sqlContext._
case class Person(name: String, age: Int)
createParquetFile[Person]("path/to/file.parquet").registerAsTable("people")
sql("INSERT INTO people SELECT 'michael', 29")
path
- The path where the directory containing parquet metadata should be created.
Data inserted into this table will also be stored at this location.allowExisting
- When false, an exception will be thrown if this directory already exists.conf
- A Hadoop configuration object that can be used to specify options to the parquet
output format.
public void registerRDDAsTable(SchemaRDD rdd, String tableName)
public SchemaRDD sql(String sqlText)
public SchemaRDD table(String tableName)
public void cacheTable(String tableName)
public void uncacheTable(String tableName)
public boolean isCached(String tableName)