Class

org.apache.spark.sql.hive

HiveContext

Related Doc: package hive

Permalink

class HiveContext extends SQLContext with Logging

An instance of the Spark SQL execution engine that integrates with data stored in Hive. Configuration for Hive is read from hive-site.xml on the classpath.

Self Type
HiveContext
Annotations
@deprecated
Deprecated

(Since version 2.0.0) Use SparkSession.builder.enableHiveSupport instead

Source
HiveContext.scala
Linear Supertypes
SQLContext, Serializable, Serializable, Logging, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. HiveContext
  2. SQLContext
  3. Serializable
  4. Serializable
  5. Logging
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HiveContext(sc: JavaSparkContext)

    Permalink
  2. new HiveContext(sc: SparkContext)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

    Permalink

    Apply a schema defined by the schema to an RDD.

    Apply a schema defined by the schema to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  5. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

    Permalink

    Apply a schema defined by the schemaString to an RDD.

    Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

    Permalink

    Convert a BaseRelation created for external data sources into a DataFrame.

    Convert a BaseRelation created for external data sources into a DataFrame.

    Definition Classes
    SQLContext
    Since

    1.3.0

  8. def cacheManager: CacheManager

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  9. def cacheTable(tableName: String): Unit

    Permalink

    Caches the specified table in-memory.

    Caches the specified table in-memory.

    Definition Classes
    SQLContext
    Since

    1.3.0

  10. def clearCache(): Unit

    Permalink

    Removes all cached tables from the in-memory cache.

    Removes all cached tables from the in-memory cache.

    Definition Classes
    SQLContext
    Since

    1.3.0

  11. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  12. def conf: SQLConf

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  13. def createDataFrame(data: List[_], beanClass: Class[_]): DataFrame

    Permalink

    Applies a schema to an List of Java Beans.

    Applies a schema to an List of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.6.0

  14. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Permalink

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.3.0

  15. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Permalink

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Since

    1.3.0

  16. def createDataFrame(rows: List[Row], schema: StructType): DataFrame

    Permalink

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an java.util.List containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided List matches the provided schema. Otherwise, there will be runtime exception.

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.6.0

  17. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    Permalink

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.3.0

  18. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    Permalink

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val dataFrame = sqlContext.createDataFrame(people, schema)
    dataFrame.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
    dataFrame.createOrReplaceTempView("people")
    sqlContext.sql("select name from people").collect.foreach(println)
    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
    Since

    1.3.0

  19. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    Permalink

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  20. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    Permalink

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.g.

    :: Experimental :: Creates a DataFrame from an RDD of Product (e.g. case classes, tuples).

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  21. def createDataset[T](data: List[T])(implicit arg0: Encoder[T]): Dataset[T]

    Permalink
    Definition Classes
    SQLContext
  22. def createDataset[T](data: RDD[T])(implicit arg0: Encoder[T]): Dataset[T]

    Permalink
    Definition Classes
    SQLContext
  23. def createDataset[T](data: Seq[T])(implicit arg0: Encoder[T]): Dataset[T]

    Permalink
    Definition Classes
    SQLContext
  24. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  25. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  26. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  27. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  28. def createExternalTable(tableName: String, path: String, source: String): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  29. def createExternalTable(tableName: String, path: String): DataFrame

    Permalink

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  30. def dropTempTable(tableName: String): Unit

    Permalink

    Drops the temporary table with the given table name in the catalog.

    Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.

    tableName

    the name of the table to be unregistered.

    Definition Classes
    SQLContext
    Since

    1.3.0

  31. def emptyDataFrame: DataFrame

    Permalink

    :: Experimental :: Returns a DataFrame with no rows or columns.

    :: Experimental :: Returns a DataFrame with no rows or columns.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  32. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  33. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  34. def executePlan(plan: LogicalPlan): QueryExecution

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  35. def executeSql(sql: String): QueryExecution

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  36. def experimental: ExperimentalMethods

    Permalink

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    Definition Classes
    SQLContext
    Annotations
    @Experimental() @transient()
    Since

    1.3.0

  37. def externalCatalog: ExternalCatalog

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  38. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  39. def getAllConfs: Map[String, String]

    Permalink

    Return all the configuration properties that have been set (i.e.

    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

    Definition Classes
    SQLContext
    Since

    1.0.0

  40. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  41. def getConf(key: String, defaultValue: String): String

    Permalink

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

    Definition Classes
    SQLContext
    Since

    1.0.0

  42. def getConf(key: String): String

    Permalink

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key.

    Definition Classes
    SQLContext
    Since

    1.0.0

  43. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  44. object implicits extends SQLImplicits with Serializable

    Permalink

    :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

    :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

    val sqlContext = new SQLContext(sc)
    import sqlContext.implicits._
    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.3.0

  45. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  46. def isCached(tableName: String): Boolean

    Permalink

    Returns true if the table is currently cached in-memory.

    Returns true if the table is currently cached in-memory.

    Definition Classes
    SQLContext
    Since

    1.3.0

  47. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  48. val isRootContext: Boolean

    Permalink
    Definition Classes
    SQLContext
  49. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  50. def listener: SQLListener

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  51. def listenerManager: ExecutionListenerManager

    Permalink

    An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

    An interface to register custom org.apache.spark.sql.util.QueryExecutionListeners that listen for execution metrics.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  52. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  53. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  54. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  55. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  56. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  57. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  59. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  65. def newSession(): HiveContext

    Permalink

    Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.

    Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.

    Definition Classes
    HiveContextSQLContext
  66. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  67. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  68. def parseDataType(dataTypeString: String): DataType

    Permalink

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  69. def parseSql(sql: String): LogicalPlan

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  70. def range(start: Long, end: Long, step: Long, numPartitions: Int): Dataset[Long]

    Permalink

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value, with partition number specified.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    2.0.0

  71. def range(start: Long, end: Long, step: Long): Dataset[Long]

    Permalink

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with an step value.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    2.0.0

  72. def range(start: Long, end: Long): Dataset[Long]

    Permalink

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from start to end (exclusive) with step value 1.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    2.0.0

  73. def range(end: Long): Dataset[Long]

    Permalink

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    :: Experimental :: Creates a Dataset with a single LongType column named id, containing elements in an range from 0 to end (exclusive) with step value 1.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    2.0.0

  74. def read: DataFrameReader

    Permalink

    :: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.

    :: Experimental :: Returns a DataFrameReader that can be used to read data and streams in as a DataFrame.

    sqlContext.read.parquet("/path/to/file.parquet")
    sqlContext.read.schema(schema).json("/path/to/file.json")
    Definition Classes
    SQLContext
    Annotations
    @Experimental()
    Since

    1.4.0

  75. def runtimeConf: RuntimeConfig

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  76. def sessionState: HiveSessionState

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  77. def setConf(key: String, value: String): Unit

    Permalink

    Set the given Spark SQL configuration property.

    Set the given Spark SQL configuration property.

    Definition Classes
    SQLContext
    Since

    1.0.0

  78. def setConf(props: Properties): Unit

    Permalink

    Set Spark SQL configuration properties.

    Set Spark SQL configuration properties.

    Definition Classes
    SQLContext
    Since

    1.0.0

  79. def sharedState: HiveSharedState

    Permalink
    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    HiveContextSQLContext
  80. def sparkContext: SparkContext

    Permalink
    Definition Classes
    SQLContext
  81. val sparkSession: SparkSession

    Permalink
    Definition Classes
    SQLContext
  82. def sql(sqlText: String): DataFrame

    Permalink

    Executes a SQL query using Spark, returning the result as a DataFrame.

    Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

    Definition Classes
    SQLContext
    Since

    1.3.0

  83. def streams: ContinuousQueryManager

    Permalink

    Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this context.

    Returns a ContinuousQueryManager that allows managing all the ContinuousQueries active on this context.

    Definition Classes
    SQLContext
    Since

    2.0.0

  84. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  85. def table(tableName: String): DataFrame

    Permalink

    Returns the specified table as a DataFrame.

    Returns the specified table as a DataFrame.

    Definition Classes
    SQLContext
    Since

    1.3.0

  86. def tableNames(databaseName: String): Array[String]

    Permalink

    Returns the names of tables in the given database as an array.

    Returns the names of tables in the given database as an array.

    Definition Classes
    SQLContext
    Since

    1.3.0

  87. def tableNames(): Array[String]

    Permalink

    Returns the names of tables in the current database as an array.

    Returns the names of tables in the current database as an array.

    Definition Classes
    SQLContext
    Since

    1.3.0

  88. def tables(databaseName: String): DataFrame

    Permalink

    Returns a DataFrame containing names of existing tables in the given database.

    Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
    Since

    1.3.0

  89. def tables(): DataFrame

    Permalink

    Returns a DataFrame containing names of existing tables in the current database.

    Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
    Since

    1.3.0

  90. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  91. def udf: UDFRegistration

    Permalink

    A collection of methods for registering user-defined functions (UDF).

    A collection of methods for registering user-defined functions (UDF).

    The following example registers a Scala closure as UDF:

    sqlContext.udf.register("myUDF", (arg1: Int, arg2: String) => arg2 + arg1)

    The following example registers a UDF in Java:

    sqlContext.udf().register("myUDF",
        new UDF2<Integer, String, String>() {
            @Override
            public String call(Integer arg1, String arg2) {
                return arg2 + arg1;
            }
       }, DataTypes.StringType);

    Or, to use Java 8 lambda syntax:

    sqlContext.udf().register("myUDF",
        (Integer arg1, String arg2) -> arg2 + arg1,
        DataTypes.StringType);
    Definition Classes
    SQLContext
    Since

    1.3.0

  92. def uncacheTable(tableName: String): Unit

    Permalink

    Removes the specified table from the in-memory cache.

    Removes the specified table from the in-memory cache.

    Definition Classes
    SQLContext
    Since

    1.3.0

  93. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  94. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  95. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from SQLContext

Inherited from Serializable

Inherited from Serializable

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Basic Operations

Cached Table Management

Configuration

Custom DataFrame Creation

Custom DataFrame Creation

Persistent Catalog DDL

Generic Data Sources

Support functions for language integrated queries