org.apache.spark.sql

SQLContext

class SQLContext extends Logging with SQLConf with ExpressionConversions with UDFRegistration with Serializable

:: AlphaComponent :: The entry point for running relational queries using Spark. Allows the creation of SchemaRDD objects and the execution of SQL queries.

Self Type
SQLContext
Annotations
@AlphaComponent()
Linear Supertypes
Serializable, Serializable, UDFRegistration, ExpressionConversions, SQLConf, Logging, AnyRef, Any
Known Subclasses
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SQLContext
  2. Serializable
  3. Serializable
  4. UDFRegistration
  5. ExpressionConversions
  6. SQLConf
  7. Logging
  8. AnyRef
  9. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SQLContext(sparkContext: SparkContext)

Type Members

  1. implicit class DslAttribute extends AnyRef

    Definition Classes
    ExpressionConversions
  2. implicit class DslExpression extends ImplicitOperators

    Definition Classes
    ExpressionConversions
  3. implicit class DslString extends ImplicitOperators

    Definition Classes
    ExpressionConversions
  4. implicit class DslSymbol extends ImplicitAttribute

    Definition Classes
    ExpressionConversions
  5. abstract class ImplicitAttribute extends ImplicitOperators

    Definition Classes
    ExpressionConversions
  6. abstract class QueryExecution extends AnyRef

    :: DeveloperApi :: The primary workflow for executing relational queries using Spark.

  7. class SparkPlanner extends SparkStrategies

    Attributes
    protected[org.apache.spark.sql]

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. lazy val analyzer: Analyzer

    Attributes
    protected[org.apache.spark.sql]
  7. def applySchema(rowRDD: RDD[Row], schema: StructType): SchemaRDD

    :: DeveloperApi :: Creates a SchemaRDD from an RDD containing Rows by applying a schema to this RDD.

    :: DeveloperApi :: Creates a SchemaRDD from an RDD containing Rows by applying a schema to this RDD. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val peopleSchemaRDD = sqlContext. applySchema(people, schema)
    peopleSchemaRDD.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
      peopleSchemaRDD.registerTempTable("people")
    sqlContext.sql("select name from people").collect.foreach(println)
    Annotations
    @DeveloperApi()
  8. def approxCountDistinct(e: Expression, rsd: Double): ApproxCountDistinct

    Definition Classes
    ExpressionConversions
  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  10. def avg(e: Expression): Average

    Definition Classes
    ExpressionConversions
  11. implicit def binaryToLiteral(a: Array[Byte]): Literal

    Definition Classes
    ExpressionConversions
  12. implicit def booleanToLiteral(b: Boolean): Literal

    Definition Classes
    ExpressionConversions
  13. implicit def byteToLiteral(b: Byte): Literal

    Definition Classes
    ExpressionConversions
  14. def cacheTable(tableName: String): Unit

    Caches the specified table in-memory.

  15. lazy val catalog: Catalog

    Attributes
    protected[org.apache.spark.sql]
  16. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  17. def count(e: Expression): Count

    Definition Classes
    ExpressionConversions
  18. def countDistinct(e: Expression*): CountDistinct

    Definition Classes
    ExpressionConversions
  19. def createParquetFile[A <: Product](path: String, allowExisting: Boolean = true, conf: Configuration = new Configuration())(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): SchemaRDD

    :: Experimental :: Creates an empty parquet file with the schema of class A, which can be registered as a table.

    :: Experimental :: Creates an empty parquet file with the schema of class A, which can be registered as a table. This registered table can be used as the target of future insertInto operations.

    val sqlContext = new SQLContext(...)
    import sqlContext._
    
    case class Person(name: String, age: Int)
    createParquetFile[Person]("path/to/file.parquet").registerTempTable("people")
    sql("INSERT INTO people SELECT 'michael', 29")
    A

    A case class type that describes the desired schema of the parquet file to be created.

    path

    The path where the directory containing parquet metadata should be created. Data inserted into this table will also be stored at this location.

    allowExisting

    When false, an exception will be thrown if this directory already exists.

    conf

    A Hadoop configuration object that can be used to specify options to the parquet output format.

    Annotations
    @Experimental()
  20. implicit def createSchemaRDD[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): SchemaRDD

    Creates a SchemaRDD from an RDD of case classes.

  21. implicit def decimalToLiteral(d: BigDecimal): Literal

    Definition Classes
    ExpressionConversions
  22. implicit def doubleToLiteral(d: Double): Literal

    Definition Classes
    ExpressionConversions
  23. lazy val emptyResult: RDD[Row]

    Attributes
    protected[org.apache.spark.sql]
  24. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  25. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  26. def executePlan(plan: LogicalPlan): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
  27. def executeSql(sql: String): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
  28. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. def first(e: Expression): First

    Definition Classes
    ExpressionConversions
  30. implicit def floatToLiteral(f: Float): Literal

    Definition Classes
    ExpressionConversions
  31. lazy val functionRegistry: FunctionRegistry

    Attributes
    protected[org.apache.spark.sql]
  32. def getAllConfs: Map[String, String]

    Return all the configuration properties that have been set (i.

    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

    Definition Classes
    SQLConf
  33. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  34. def getConf(key: String, defaultValue: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

    Definition Classes
    SQLConf
  35. def getConf(key: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key.

    Definition Classes
    SQLConf
  36. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  37. implicit def intToLiteral(i: Int): Literal

    Definition Classes
    ExpressionConversions
  38. def isCached(tableName: String): Boolean

    Returns true if the table is currently cached in-memory.

  39. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  40. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  41. def jsonFile(path: String, samplingRatio: Double): SchemaRDD

    :: Experimental ::

    :: Experimental ::

    Annotations
    @Experimental()
  42. def jsonFile(path: String, schema: StructType): SchemaRDD

    :: Experimental :: Loads a JSON file (one object per line) and applies the given schema, returning the result as a SchemaRDD.

    :: Experimental :: Loads a JSON file (one object per line) and applies the given schema, returning the result as a SchemaRDD.

    Annotations
    @Experimental()
  43. def jsonFile(path: String): SchemaRDD

    Loads a JSON file (one object per line), returning the result as a SchemaRDD.

    Loads a JSON file (one object per line), returning the result as a SchemaRDD. It goes through the entire dataset once to determine the schema.

  44. def jsonRDD(json: RDD[String], samplingRatio: Double): SchemaRDD

    :: Experimental ::

    :: Experimental ::

    Annotations
    @Experimental()
  45. def jsonRDD(json: RDD[String], schema: StructType): SchemaRDD

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a SchemaRDD.

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a SchemaRDD.

    Annotations
    @Experimental()
  46. def jsonRDD(json: RDD[String]): SchemaRDD

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a SchemaRDD.

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a SchemaRDD. It goes through the entire dataset once to determine the schema.

  47. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  48. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  49. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  50. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  51. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  52. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  53. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  54. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  55. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  56. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  57. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  58. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  59. implicit def logicalPlanToSparkQuery(plan: LogicalPlan): SchemaRDD

    :: DeveloperApi :: Allows catalyst LogicalPlans to be executed as a SchemaRDD.

    :: DeveloperApi :: Allows catalyst LogicalPlans to be executed as a SchemaRDD. Note that the LogicalPlan interface is considered internal, and thus not guaranteed to be stable. As a result, using them directly is not recommended.

    Annotations
    @DeveloperApi()
  60. implicit def longToLiteral(l: Long): Literal

    Definition Classes
    ExpressionConversions
  61. def lower(e: Expression): Lower

    Definition Classes
    ExpressionConversions
  62. def max(e: Expression): Max

    Definition Classes
    ExpressionConversions
  63. def min(e: Expression): Min

    Definition Classes
    ExpressionConversions
  64. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  65. final def notify(): Unit

    Definition Classes
    AnyRef
  66. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  67. val optimizer: Optimizer.type

    Attributes
    protected[org.apache.spark.sql]
  68. def parquetFile(path: String): SchemaRDD

    Loads a Parquet file, returning the result as a SchemaRDD.

  69. def parseSql(sql: String): LogicalPlan

    Attributes
    protected[org.apache.spark.sql]
  70. val parser: SqlParser

    Attributes
    protected[org.apache.spark.sql]
  71. val planner: SparkPlanner

    Attributes
    protected[org.apache.spark.sql]
  72. val prepareForExecution: RuleExecutor[SparkPlan] { val batches: List[this.Batch] }

    Prepares a planned SparkPlan for execution by inserting shuffle operations as needed.

    Prepares a planned SparkPlan for execution by inserting shuffle operations as needed.

    Attributes
    protected[org.apache.spark.sql]
  73. def registerFunction[T](name: String, func: Function22[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  74. def registerFunction[T](name: String, func: Function21[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  75. def registerFunction[T](name: String, func: Function20[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  76. def registerFunction[T](name: String, func: Function19[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  77. def registerFunction[T](name: String, func: Function18[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  78. def registerFunction[T](name: String, func: Function17[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  79. def registerFunction[T](name: String, func: Function16[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  80. def registerFunction[T](name: String, func: Function15[_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  81. def registerFunction[T](name: String, func: Function14[_, _, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  82. def registerFunction[T](name: String, func: Function13[_, _, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  83. def registerFunction[T](name: String, func: Function12[_, _, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  84. def registerFunction[T](name: String, func: Function11[_, _, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  85. def registerFunction[T](name: String, func: Function10[_, _, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  86. def registerFunction[T](name: String, func: Function9[_, _, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  87. def registerFunction[T](name: String, func: Function8[_, _, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  88. def registerFunction[T](name: String, func: Function7[_, _, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  89. def registerFunction[T](name: String, func: Function6[_, _, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  90. def registerFunction[T](name: String, func: Function5[_, _, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  91. def registerFunction[T](name: String, func: Function4[_, _, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  92. def registerFunction[T](name: String, func: Function3[_, _, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  93. def registerFunction[T](name: String, func: Function2[_, _, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    Definition Classes
    UDFRegistration
  94. def registerFunction[T](name: String, func: Function1[_, T])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T]): Unit

    registerFunction 1-22 were generated by this script

    registerFunction 1-22 were generated by this script

    (1 to 22).map { x => val types = (1 to x).map(x => "_").reduce(_ + ", " + _) s""" def registerFunction[T: TypeTag](name: String, func: Function$x[$types, T]): Unit = { def builder(e: Seq[Expression]) = ScalaUdf(func, ScalaReflection.schemaFor(typeTag[T]).dataType, e) functionRegistry.registerFunction(name, builder) } """ }

    Definition Classes
    UDFRegistration
  95. def registerRDDAsTable(rdd: SchemaRDD, tableName: String): Unit

    Registers the given RDD as a temporary table in the catalog.

    Registers the given RDD as a temporary table in the catalog. Temporary tables exist only during the lifetime of this instance of SQLContext.

  96. def setConf(key: String, value: String): Unit

    Set the given Spark SQL configuration property.

    Set the given Spark SQL configuration property.

    Definition Classes
    SQLConf
  97. def setConf(props: Properties): Unit

    Set Spark SQL configuration properties.

    Set Spark SQL configuration properties.

    Definition Classes
    SQLConf
  98. val settings: Map[String, String]

    Only low degree of contention is expected for conf, thus NOT using ConcurrentHashMap.

    Only low degree of contention is expected for conf, thus NOT using ConcurrentHashMap.

    Attributes
    protected[org.apache.spark]
    Definition Classes
    SQLConf
  99. implicit def shortToLiteral(s: Short): Literal

    Definition Classes
    ExpressionConversions
  100. val sparkContext: SparkContext

  101. def sql(sqlText: String): SchemaRDD

    Executes a SQL query using Spark, returning the result as a SchemaRDD.

    Executes a SQL query using Spark, returning the result as a SchemaRDD. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

  102. implicit def stringToLiteral(s: String): Literal

    Definition Classes
    ExpressionConversions
  103. def sum(e: Expression): Sum

    Definition Classes
    ExpressionConversions
  104. def sumDistinct(e: Expression): SumDistinct

    Definition Classes
    ExpressionConversions
  105. implicit def symbolToUnresolvedAttribute(s: Symbol): UnresolvedAttribute

    Definition Classes
    ExpressionConversions
  106. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  107. def table(tableName: String): SchemaRDD

    Returns the specified table as a SchemaRDD

  108. implicit def timestampToLiteral(t: Timestamp): Literal

    Definition Classes
    ExpressionConversions
  109. def toString(): String

    Definition Classes
    AnyRef → Any
  110. def uncacheTable(tableName: String): Unit

    Removes the specified table from the in-memory cache.

  111. def upper(e: Expression): Upper

    Definition Classes
    ExpressionConversions
  112. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  113. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  114. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from UDFRegistration

Inherited from ExpressionConversions

Inherited from SQLConf

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Spark SQL Functions

Ungrouped