org.apache.spark.sql.hive

HiveContext

class HiveContext extends SQLContext with Logging

An instance of the Spark SQL execution engine that integrates with data stored in Hive. Configuration for Hive is read from hive-site.xml on the classpath.

Self Type
HiveContext
Source
HiveContext.scala
Since

1.0.0

Linear Supertypes
SQLContext, Serializable, Serializable, Logging, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By inheritance
Inherited
  1. HiveContext
  2. SQLContext
  3. Serializable
  4. Serializable
  5. Logging
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HiveContext(sc: JavaSparkContext)

  2. new HiveContext(sc: SparkContext)

Type Members

  1. class QueryExecution extends execution.QueryExecution

    Extends QueryExecution with hive specific features.

  2. class SparkPlanner extends execution.SparkPlanner

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.6.0) use org.apache.spark.sql.SparkPlanner

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. def addJar(path: String): Unit

    Add a jar to SQLContext

    Add a jar to SQLContext

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  7. def analyze(tableName: String): Unit

    Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

    Analyzes the given table in the current database to generate statistics, which will be used in query optimizations.

    Right now, it only supports Hive tables and it only updates the size of a Hive table in the Hive metastore.

    Since

    1.2.0

  8. lazy val analyzer: Analyzer

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  9. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

    Apply a schema defined by the schema to an RDD.

    Apply a schema defined by the schema to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  10. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

    Apply a schema defined by the schemaString to an RDD.

    Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  11. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  12. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

    Convert a BaseRelation created for external data sources into a DataFrame.

    Convert a BaseRelation created for external data sources into a DataFrame.

    Definition Classes
    SQLContext
    Since

    1.3.0

  13. def cacheTable(tableName: String): Unit

    Caches the specified table in-memory.

    Caches the specified table in-memory.

    Definition Classes
    SQLContext
    Since

    1.3.0

  14. lazy val catalog: HiveMetastoreCatalog with OverrideCatalog

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  15. def clearCache(): Unit

    Removes all cached tables from the in-memory cache.

    Removes all cached tables from the in-memory cache.

    Definition Classes
    SQLContext
    Since

    1.3.0

  16. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  17. lazy val conf: SQLConf

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  18. def configure(): Map[String, String]

    Overridden by child classes that need to set configuration before the client init.

    Overridden by child classes that need to set configuration before the client init.

    Attributes
    protected
  19. def convertCTAS: Boolean

    When true, a table created by a Hive CTAS statement (no USING clause) will be converted to a data source table, using the data source set by spark.

    When true, a table created by a Hive CTAS statement (no USING clause) will be converted to a data source table, using the data source set by spark.sql.sources.default. The table in CTAS statement will be converted when it meets any of the following conditions:

    • The CTAS does not specify any of a SerDe (ROW FORMAT SERDE), a File Format (STORED AS), or a Storage Hanlder (STORED BY), and the value of hive.default.fileformat in hive-site.xml is either TextFile or SequenceFile.
    • The CTAS statement specifies TextFile (STORED AS TEXTFILE) as the file format and no SerDe is specified (no ROW FORMAT SERDE clause).
    • The CTAS statement specifies SequenceFile (STORED AS SEQUENCEFILE) as the file format and no SerDe is specified (no ROW FORMAT SERDE clause).
    Attributes
    protected[org.apache.spark.sql]
  20. def convertMetastoreParquet: Boolean

    When true, enables an experimental feature where metastore tables that use the parquet SerDe are automatically converted to use the Spark SQL parquet table scan, instead of the Hive SerDe.

    When true, enables an experimental feature where metastore tables that use the parquet SerDe are automatically converted to use the Spark SQL parquet table scan, instead of the Hive SerDe.

    Attributes
    protected[org.apache.spark.sql]
  21. def convertMetastoreParquetWithSchemaMerging: Boolean

    When true, also tries to merge possibly different but compatible Parquet schemas in different Parquet data files.

    When true, also tries to merge possibly different but compatible Parquet schemas in different Parquet data files.

    This configuration is only effective when "spark.sql.hive.convertMetastoreParquet" is true.

    Attributes
    protected[org.apache.spark.sql]
  22. def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

    Applies a schema to an List of Java Beans.

    Applies a schema to an List of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.6.0

  23. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.3.0

  24. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.3.0

  25. def createDataFrame(rows: List[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.6.0

  26. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.3.0

  27. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val dataFrame = sqlContext.createDataFrame(people, schema)
    dataFrame.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
    dataFrame.registerTempTable("people")
    sqlContext.sql("select name from people").collect.foreach(println)
    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.3.0

  28. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  29. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  30. def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

    Definition Classes
    SQLContext
  31. def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

    Definition Classes
    SQLContext
  32. def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

    Definition Classes
    SQLContext
  33. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  34. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  35. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  36. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  37. def createExternalTable(tableName: String, path: String, source: String): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  38. def createExternalTable(tableName: String, path: String): DataFrame

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  39. val ddlParser: DDLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  40. def dialectClassName: String

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  41. def dropTempTable(tableName: String): Unit

    Drops the temporary table with the given table name in the catalog.

    Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.

    tableName

    the name of the table to be unregistered.

    Definition Classes
    SQLContext
    Since

    1.3.0

  42. lazy val emptyDataFrame: DataFrame

    :: Experimental :: Returns a DataFrame with no rows or columns.

    :: Experimental :: Returns a DataFrame with no rows or columns.

    Definition Classes
    SQLContext
    Since

    1.3.0

  43. lazy val emptyResult: RDD[InternalRow]

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  44. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  45. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  46. def executePlan(plan: LogicalPlan): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  47. def executeSql(sql: String): execution.QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  48. lazy val executionHive: ClientWrapper

    The copy of the hive client that is used for execution.

    The copy of the hive client that is used for execution. Currently this must always be Hive 13 as this is the version of Hive that is packaged with Spark SQL. This copy of the client is used for execution related tasks like registering temporary functions or ensuring that the ThreadLocal SessionState is correctly populated. This copy of Hive is *not* used for storing persistent metadata, and only point to a dummy metastore in a temporary directory.

    Attributes
    protected[org.apache.spark.sql.hive]
  49. val experimental: ExperimentalMethods

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    Definition Classes
    SQLContext
    Since

    1.3.0

  50. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  51. lazy val functionRegistry: FunctionRegistry

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  52. def getAllConfs: Map[String, String]

    Return all the configuration properties that have been set (i.

    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

    Definition Classes
    SQLContext
    Since

    1.0.0

  53. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  54. def getConf(key: String, defaultValue: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

    Definition Classes
    SQLContext
    Since

    1.0.0

  55. def getConf(key: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key.

    Definition Classes
    SQLContext
    Since

    1.0.0

  56. def getSQLDialect(): ParserDialect

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  57. def getSchema(beanClass: Class[_]): Seq[AttributeReference]

    Returns a Catalyst Schema for the given java bean class.

    Returns a Catalyst Schema for the given java bean class.

    Attributes
    protected
    Definition Classes
    SQLContext
  58. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  59. def hiveMetastoreBarrierPrefixes: Seq[String]

    A comma separated list of class prefixes that should explicitly be reloaded for each version of Hive that Spark SQL is communicating with.

    A comma separated list of class prefixes that should explicitly be reloaded for each version of Hive that Spark SQL is communicating with. For example, Hive UDFs that are declared in a prefix that typically would be shared (i.e. org.apache.spark.*)

    Attributes
    protected[org.apache.spark.sql.hive]
  60. def hiveMetastoreJars: String

    The location of the jars that should be used to instantiate the HiveMetastoreClient.

    The location of the jars that should be used to instantiate the HiveMetastoreClient. This property can be one of three options:

    • a classpath in the standard format for both hive and hadoop.
    • builtin - attempt to discover the jars that were used to load Spark SQL and use those. This option is only valid when using the execution version of Hive.
    • maven - download the correct version of hive on demand from maven.
    Attributes
    protected[org.apache.spark.sql.hive]
  61. def hiveMetastoreSharedPrefixes: Seq[String]

    A comma separated list of class prefixes that should be loaded using the classloader that is shared between Spark SQL and a specific version of Hive.

    A comma separated list of class prefixes that should be loaded using the classloader that is shared between Spark SQL and a specific version of Hive. An example of classes that should be shared is JDBC drivers that are needed to talk to the metastore. Other classes that need to be shared are those that interact with classes that are already shared. For example, custom appenders that are used by log4j.

    Attributes
    protected[org.apache.spark.sql.hive]
  62. def hiveMetastoreVersion: String

    The version of the hive client that will be used to communicate with the metastore.

    The version of the hive client that will be used to communicate with the metastore. Note that this does not necessarily need to be the same version of Hive that is used internally by Spark SQL for execution.

    Attributes
    protected[org.apache.spark.sql.hive]
  63. def hiveThriftServerAsync: Boolean

    Attributes
    protected[org.apache.spark.sql.hive]
  64. def hiveThriftServerSingleSession: Boolean

    Attributes
    protected[org.apache.spark.sql.hive]
  65. lazy val hiveconf: HiveConf

    SQLConf and HiveConf contracts:

    SQLConf and HiveConf contracts:

    1. create a new SessionState for each HiveContext 2. when the Hive session is first initialized, params in HiveConf will get picked up by the SQLConf. Additionally, any properties set by set() or a SET command inside sql() will be set in the SQLConf *as well as* in the HiveConf.

    Attributes
    protected[org.apache.spark.sql.hive]
  66. object implicits extends SQLImplicits with Serializable

    :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

  67. def invalidateTable(tableName: String): Unit

    Attributes
    protected[org.apache.spark.sql.hive]
  68. def isCached(tableName: String): Boolean

    Returns true if the table is currently cached in-memory.

    Returns true if the table is currently cached in-memory.

    Definition Classes
    SQLContext
    Since

    1.3.0

  69. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  70. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  71. lazy val listenerManager: ExecutionListenerManager

    Definition Classes
    SQLContext
  72. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  73. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  74. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  75. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  76. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  77. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  78. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  79. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  80. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  81. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  82. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  83. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  84. lazy val metadataHive: ClientInterface

    The copy of the Hive client that is used to retrieve metadata from the Hive MetaStore.

    The copy of the Hive client that is used to retrieve metadata from the Hive MetaStore. The version of the Hive client that is used here must match the metastore that is configured in the hive-site.xml file.

    Attributes
    protected[org.apache.spark.sql.hive]
  85. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  86. def newSession(): HiveContext

    Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.

    Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.

    Definition Classes
    HiveContextSQLContext
  87. final def notify(): Unit

    Definition Classes
    AnyRef
  88. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  89. lazy val optimizer: Optimizer

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  90. def parseDataType(dataTypeString: String): DataType

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  91. def parseSql(sql: String): LogicalPlan

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  92. val planner: SparkPlanner with HiveStrategies

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  93. val prepareForExecution: RuleExecutor[SparkPlan]

    Prepares a planned SparkPlan for execution by inserting shuffle operations and internal row format conversions as needed.

    Prepares a planned SparkPlan for execution by inserting shuffle operations and internal row format conversions as needed.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  94. def range(start: Long, end: Long, step: Long, numPartitions: Int): DataFrame

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.4.0

  95. def range(start: Long, end: Long): DataFrame

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.4.0

  96. def range(end: Long): DataFrame

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    :: Experimental :: Creates a DataFrame with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.4.1

  97. def read: DataFrameReader

    :: Experimental :: Returns a DataFrameReader that can be used to read data in as a DataFrame.

    :: Experimental :: Returns a DataFrameReader that can be used to read data in as a DataFrame.

    sqlContext.read.parquet("/path/to/file.parquet")
    sqlContext.read.schema(schema).json("/path/to/file.json")
    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.4.0

  98. def refreshTable(tableName: String): Unit

    Invalidate and refresh all the cached the metadata of the given table.

    Invalidate and refresh all the cached the metadata of the given table. For performance reasons, Spark SQL or the external data source library it uses might cache certain metadata about a table, such as the location of blocks. When those change outside of Spark SQL, users should call this function to invalidate the cache.

    Since

    1.3.0

  99. def runSqlHive(sql: String): Seq[String]

    Attributes
    protected[org.apache.spark.sql.hive]
  100. def setConf(key: String, value: String): Unit

    Set the given Spark SQL configuration property.

    Set the given Spark SQL configuration property.

    Definition Classes
    HiveContextSQLContext
    Since

    1.0.0

  101. def setConf(props: Properties): Unit

    Set Spark SQL configuration properties.

    Set Spark SQL configuration properties.

    Definition Classes
    SQLContext
    Since

    1.0.0

  102. val sparkContext: SparkContext

    Definition Classes
    SQLContext
  103. def sql(sqlText: String): DataFrame

    Executes a SQL query using Spark, returning the result as a DataFrame.

    Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

    Definition Classes
    SQLContext
    Since

    1.3.0

  104. val sqlParser: SparkSQLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  105. lazy val substitutor: VariableSubstitution

    Attributes
    protected[org.apache.spark.sql]
  106. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  107. def table(tableName: String): DataFrame

    Returns the specified table as a DataFrame.

    Returns the specified table as a DataFrame.

    Definition Classes
    SQLContext
    Since

    1.3.0

  108. def tableNames(databaseName: String): Array[String]

    Returns the names of tables in the given database as an array.

    Returns the names of tables in the given database as an array.

    Definition Classes
    SQLContext
    Since

    1.3.0

  109. def tableNames(): Array[String]

    Returns the names of tables in the current database as an array.

    Returns the names of tables in the current database as an array.

    Definition Classes
    SQLContext
    Since

    1.3.0

  110. def tables(databaseName: String): DataFrame

    Returns a DataFrame containing names of existing tables in the given database.

    Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
    Since

    1.3.0

  111. def tables(): DataFrame

    Returns a DataFrame containing names of existing tables in the current database.

    Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
    Since

    1.3.0

  112. def toString(): String

    Definition Classes
    AnyRef → Any
  113. val udf: UDFRegistration

    A collection of methods for registering user-defined functions (UDF).

    A collection of methods for registering user-defined functions (UDF).

    The following example registers a Scala closure as UDF:

    sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

    The following example registers a UDF in Java:

    sqlContext.udf().register("myUDF",
     new UDF2<Integer, String, String>() {
         @Override
         public String call(Integer arg1, String arg2) {
             return arg2 + arg1;
         }
    }, DataTypes.StringType);

    Or, to use Java 8 lambda syntax:

    sqlContext.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1,
    DataTypes.StringType);
    Definition Classes
    SQLContext
    Since

    1.3.0 TODO move to SQLSession?

  114. def uncacheTable(tableName: String): Unit

    Removes the specified table from the in-memory cache.

    Removes the specified table from the in-memory cache.

    Definition Classes
    SQLContext
    Since

    1.3.0

  115. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  116. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  117. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def applySchema(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use createDataFrame. This will be removed in Spark 2.0.

  2. def applySchema(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use createDataFrame. This will be removed in Spark 2.0.

  3. def applySchema(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use createDataFrame. This will be removed in Spark 2.0.

  4. def applySchema(rowRDD: RDD[Row], schema: StructType): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) Use createDataFrame. This will be removed in Spark 2.0.

  5. def jdbc(url: String, table: String, theParts: Array[String]): DataFrame

    Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    Construct a DataFrame representing the database table accessible via JDBC URL url named table. The theParts parameter gives a list expressions suitable for inclusion in WHERE clauses; each one defines one partition of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.jdbc(). This will be removed in Spark 2.0.

  6. def jdbc(url: String, table: String, columnName: String, lowerBound: Long, upperBound: Long, numPartitions: Int): DataFrame

    Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    Construct a DataFrame representing the database table accessible via JDBC URL url named table. Partitions of the table will be retrieved in parallel based on the parameters passed to this function.

    columnName

    the name of a column of integral type that will be used for partitioning.

    lowerBound

    the minimum value of columnName used to decide partition stride

    upperBound

    the maximum value of columnName used to decide partition stride

    numPartitions

    the number of partitions. the range minValue-maxValue will be split evenly into this many partitions

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.jdbc(). This will be removed in Spark 2.0.

  7. def jdbc(url: String, table: String): DataFrame

    Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.jdbc(). This will be removed in Spark 2.0.

  8. def jsonFile(path: String, samplingRatio: Double): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  9. def jsonFile(path: String, schema: StructType): DataFrame

    Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

    Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  10. def jsonFile(path: String): DataFrame

    Loads a JSON file (one object per line), returning the result as a DataFrame.

    Loads a JSON file (one object per line), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  11. def jsonRDD(json: JavaRDD[String], samplingRatio: Double): DataFrame

    Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  12. def jsonRDD(json: RDD[String], samplingRatio: Double): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  13. def jsonRDD(json: JavaRDD[String], schema: StructType): DataFrame

    Loads an JavaRDD<String> storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Loads an JavaRDD<String> storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  14. def jsonRDD(json: RDD[String], schema: StructType): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  15. def jsonRDD(json: JavaRDD[String]): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  16. def jsonRDD(json: RDD[String]): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.json(). This will be removed in Spark 2.0.

  17. def load(source: String, schema: StructType, options: Map[String, String]): DataFrame

    (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0)

  18. def load(source: String, schema: StructType, options: Map[String, String]): DataFrame

    (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0)

  19. def load(source: String, options: Map[String, String]): DataFrame

    (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0)

  20. def load(source: String, options: Map[String, String]): DataFrame

    (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0)

  21. def load(path: String, source: String): DataFrame

    Returns the dataset stored at path as a DataFrame, using the given data source.

    Returns the dataset stored at path as a DataFrame, using the given data source.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.format(source).load(path). This will be removed in Spark 2.0.

  22. def load(path: String): DataFrame

    Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.

    Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.4.0) Use read.load(path). This will be removed in Spark 2.0.

  23. def parquetFile(paths: String*): DataFrame

    Loads a Parquet file, returning the result as a DataFrame.

    Loads a Parquet file, returning the result as a DataFrame. This function returns an empty DataFrame if no paths are passed in.

    Definition Classes
    SQLContext
    Annotations
    @deprecated @varargs()
    Deprecated

    (Since version 1.4.0) Use read.parquet(). This will be removed in Spark 2.0.

Inherited from SQLContext

Inherited from Serializable

Inherited from Serializable

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Basic Operations

Cached Table Management

Configuration

dataframe

Custom DataFrame Creation

Persistent Catalog DDL

Generic Data Sources

Specific Data Sources

Support functions for language integrated queries