org.apache.spark.sql.test

TestSQLContext

object TestSQLContext extends SQLContext

A SQLContext that can be used for local testing.

Linear Supertypes
SQLContext, Serializable, Serializable, Logging, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By inheritance
Inherited
  1. TestSQLContext
  2. SQLContext
  3. Serializable
  4. Serializable
  5. Logging
  6. AnyRef
  7. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. class QueryExecution extends AnyRef

    :: DeveloperApi :: The primary workflow for executing relational queries using Spark.

  2. class SparkPlanner extends SparkStrategies

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any
  3. final def ##(): Int

    Definition Classes
    AnyRef → Any
  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any
  6. lazy val analyzer: Analyzer

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  7. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schema: StructType): DataFrame

    Apply a schema defined by the schema to an RDD.

    Apply a schema defined by the schema to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  8. def applySchemaToPythonRDD(rdd: RDD[Array[Any]], schemaString: String): DataFrame

    Apply a schema defined by the schemaString to an RDD.

    Apply a schema defined by the schemaString to an RDD. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  9. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  10. def baseRelationToDataFrame(baseRelation: BaseRelation): DataFrame

    Convert a BaseRelation created for external data sources into a DataFrame.

    Convert a BaseRelation created for external data sources into a DataFrame.

    Definition Classes
    SQLContext
  11. val cacheManager: CacheManager

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  12. def cacheTable(tableName: String): Unit

    Caches the specified table in-memory.

    Caches the specified table in-memory.

    Definition Classes
    SQLContext
  13. lazy val catalog: Catalog

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  14. def clearCache(): Unit

    Removes all cached tables from the in-memory cache.

    Removes all cached tables from the in-memory cache.

    Definition Classes
    SQLContext
  15. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  16. lazy val conf: SQLConf

    Fewer partitions to speed up testing.

    Fewer partitions to speed up testing.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    TestSQLContextSQLContext
  17. def createDataFrame(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
  18. def createDataFrame(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
  19. def createDataFrame(rowRDD: JavaRDD[Row], columns: List[String]): DataFrame

    Creates a DataFrame from an JavaRDD containing Rows by applying a seq of names of columns to this RDD, the data type for each column will be inferred by the first row.

    Creates a DataFrame from an JavaRDD containing Rows by applying a seq of names of columns to this RDD, the data type for each column will be inferred by the first row.

    rowRDD

    an JavaRDD of Row

    columns

    names for each column

    returns

    DataFrame

    Definition Classes
    SQLContext
  20. def createDataFrame(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an JavaRDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception.

    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
  21. def createDataFrame(rowRDD: RDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema.

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows using the given schema. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val dataFrame = sqlContext.createDataFrame(people, schema)
    dataFrame.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
    dataFrame.registerTempTable("people")
    sqlContext.sql("select name from people").collect.foreach(println)
    Definition Classes
    SQLContext
    Annotations
    @DeveloperApi()
  22. def createDataFrame[A <: Product](data: Seq[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    :: Experimental :: Creates a DataFrame from a local Seq of Product.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  23. def createDataFrame[A <: Product](rdd: RDD[A])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[A]): DataFrame

    :: Experimental :: Creates a DataFrame from an RDD of case classes.

    :: Experimental :: Creates a DataFrame from an RDD of case classes.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  24. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: (Scala-specific) Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  25. def createExternalTable(tableName: String, source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options.

    :: Experimental :: Create an external table from the given path based on a data source, a schema and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  26. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: (Scala-specific) Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  27. def createExternalTable(tableName: String, source: String, options: Map[String, String]): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options.

    :: Experimental :: Creates an external table from the given path based on a data source and a set of options. Then, returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  28. def createExternalTable(tableName: String, path: String, source: String): DataFrame

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path based on a data source and returns the corresponding DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  29. def createExternalTable(tableName: String, path: String): DataFrame

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame.

    :: Experimental :: Creates an external table from the given path and returns the corresponding DataFrame. It will use the default data source configured by spark.sql.sources.default.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  30. val ddlParser: DDLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  31. def dropTempTable(tableName: String): Unit

    Drops the temporary table with the given table name in the catalog.

    Drops the temporary table with the given table name in the catalog. If the table has been cached/persisted before, it's also unpersisted.

    tableName

    the name of the table to be unregistered.

    Definition Classes
    SQLContext
  32. lazy val emptyDataFrame: DataFrame

    :: Experimental :: Returns a DataFrame with no rows or columns.

    :: Experimental :: Returns a DataFrame with no rows or columns.

    Definition Classes
    SQLContext
  33. lazy val emptyResult: RDD[Row]

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  34. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  35. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  36. def executePlan(plan: LogicalPlan): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  37. def executeSql(sql: String): QueryExecution

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  38. val experimental: ExperimentalMethods

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    :: Experimental :: A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.

    Definition Classes
    SQLContext
  39. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  40. lazy val functionRegistry: FunctionRegistry

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  41. def getAllConfs: Map[String, String]

    Return all the configuration properties that have been set (i.

    Return all the configuration properties that have been set (i.e. not the default). This creates a new copy of the config properties in the form of a Map.

    Definition Classes
    SQLContext
  42. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  43. def getConf(key: String, defaultValue: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key. If the key is not set yet, return defaultValue.

    Definition Classes
    SQLContext
  44. def getConf(key: String): String

    Return the value of Spark SQL configuration property for the given key.

    Return the value of Spark SQL configuration property for the given key.

    Definition Classes
    SQLContext
  45. def getSchema(beanClass: Class[_]): Seq[AttributeReference]

    Returns a Catalyst Schema for the given java bean class.

    Returns a Catalyst Schema for the given java bean class.

    Attributes
    protected
    Definition Classes
    SQLContext
  46. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  47. object implicits extends Serializable

    :: Experimental :: (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.

  48. def isCached(tableName: String): Boolean

    Returns true if the table is currently cached in-memory.

    Returns true if the table is currently cached in-memory.

    Definition Classes
    SQLContext
  49. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  50. def isTraceEnabled(): Boolean

    Attributes
    protected
    Definition Classes
    Logging
  51. def jdbc(url: String, table: String, theParts: Array[String]): DataFrame

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table. The theParts parameter gives a list expressions suitable for inclusion in WHERE clauses; each one defines one partition of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  52. def jdbc(url: String, table: String, columnName: String, lowerBound: Long, upperBound: Long, numPartitions: Int): DataFrame

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table. Partitions of the table will be retrieved in parallel based on the parameters passed to this function.

    columnName

    the name of a column of integral type that will be used for partitioning.

    lowerBound

    the minimum value of columnName to retrieve

    upperBound

    the maximum value of columnName to retrieve

    numPartitions

    the number of partitions. the range minValue-maxValue will be split evenly into this many partitions

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  53. def jdbc(url: String, table: String): DataFrame

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    :: Experimental :: Construct a DataFrame representing the database table accessible via JDBC URL url named table.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  54. def jsonFile(path: String, samplingRatio: Double): DataFrame

    :: Experimental ::

    :: Experimental ::

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  55. def jsonFile(path: String, schema: StructType): DataFrame

    :: Experimental :: Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

    :: Experimental :: Loads a JSON file (one object per line) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  56. def jsonFile(path: String): DataFrame

    Loads a JSON file (one object per line), returning the result as a DataFrame.

    Loads a JSON file (one object per line), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
  57. def jsonRDD(json: JavaRDD[String], samplingRatio: Double): DataFrame

    :: Experimental :: Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    :: Experimental :: Loads a JavaRDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  58. def jsonRDD(json: RDD[String], samplingRatio: Double): DataFrame

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) inferring the schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  59. def jsonRDD(json: JavaRDD[String], schema: StructType): DataFrame

    :: Experimental :: Loads an JavaRDD<String> storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    :: Experimental :: Loads an JavaRDD<String> storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  60. def jsonRDD(json: RDD[String], schema: StructType): DataFrame

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    :: Experimental :: Loads an RDD[String] storing JSON objects (one object per record) and applies the given schema, returning the result as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  61. def jsonRDD(json: JavaRDD[String]): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
  62. def jsonRDD(json: RDD[String]): DataFrame

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame.

    Loads an RDD[String] storing JSON objects (one object per record), returning the result as a DataFrame. It goes through the entire dataset once to determine the schema.

    Definition Classes
    SQLContext
  63. def load(source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    :: Experimental :: (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  64. def load(source: String, schema: StructType, options: Map[String, String]): DataFrame

    :: Experimental :: (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    :: Experimental :: (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame, using the given schema as the schema of the DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  65. def load(source: String, options: Map[String, String]): DataFrame

    :: Experimental :: (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    :: Experimental :: (Scala-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  66. def load(source: String, options: Map[String, String]): DataFrame

    :: Experimental :: (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    :: Experimental :: (Java-specific) Returns the dataset specified by the given data source and a set of options as a DataFrame.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  67. def load(path: String, source: String): DataFrame

    :: Experimental :: Returns the dataset stored at path as a DataFrame, using the given data source.

    :: Experimental :: Returns the dataset stored at path as a DataFrame, using the given data source.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  68. def load(path: String): DataFrame

    :: Experimental :: Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.

    :: Experimental :: Returns the dataset stored at path as a DataFrame, using the default data source configured by spark.sql.sources.default.

    Definition Classes
    SQLContext
    Annotations
    @Experimental()
  69. def log: Logger

    Attributes
    protected
    Definition Classes
    Logging
  70. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  71. def logDebug(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  72. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  73. def logError(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  74. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  75. def logInfo(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  76. def logName: String

    Attributes
    protected
    Definition Classes
    Logging
  77. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  78. def logTrace(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  79. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Attributes
    protected
    Definition Classes
    Logging
  80. def logWarning(msg: ⇒ String): Unit

    Attributes
    protected
    Definition Classes
    Logging
  81. implicit def logicalPlanToSparkQuery(plan: LogicalPlan): DataFrame

    Turn a logical plan into a DataFrame.

    Turn a logical plan into a DataFrame. This should be removed once we have an easier way to construct DataFrame directly out of local data without relying on implicits.

    Attributes
    protected[org.apache.spark.sql]
  82. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  83. final def notify(): Unit

    Definition Classes
    AnyRef
  84. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  85. lazy val optimizer: Optimizer

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  86. def parquetFile(paths: String*): DataFrame

    Loads a Parquet file, returning the result as a DataFrame.

    Loads a Parquet file, returning the result as a DataFrame. This function returns an empty DataFrame if no paths are passed in.

    Definition Classes
    SQLContext
    Annotations
    @varargs()
  87. def parseDataType(dataTypeString: String): DataType

    Parses the data type in our internal string representation.

    Parses the data type in our internal string representation. The data type string should have the same format as the one generated by toString in scala. It is only used by PySpark.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  88. def parseSql(sql: String): LogicalPlan

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  89. val planner: SparkPlanner

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  90. val prepareForExecution: RuleExecutor[SparkPlan] { val batches: List[this.Batch] }

    Prepares a planned SparkPlan for execution by inserting shuffle operations as needed.

    Prepares a planned SparkPlan for execution by inserting shuffle operations as needed.

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  91. def setConf(key: String, value: String): Unit

    Set the given Spark SQL configuration property.

    Set the given Spark SQL configuration property.

    Definition Classes
    SQLContext
  92. def setConf(props: Properties): Unit

    Set Spark SQL configuration properties.

    Set Spark SQL configuration properties.

    Definition Classes
    SQLContext
  93. val sparkContext: SparkContext

    Definition Classes
    SQLContext
  94. def sql(sqlText: String): DataFrame

    Executes a SQL query using Spark, returning the result as a DataFrame.

    Executes a SQL query using Spark, returning the result as a DataFrame. The dialect that is used for SQL parsing can be configured with 'spark.sql.dialect'.

    Definition Classes
    SQLContext
  95. val sqlParser: SparkSQLParser

    Attributes
    protected[org.apache.spark.sql]
    Definition Classes
    SQLContext
  96. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  97. def table(tableName: String): DataFrame

    Returns the specified table as a DataFrame.

    Returns the specified table as a DataFrame.

    Definition Classes
    SQLContext
  98. def tableNames(databaseName: String): Array[String]

    Returns the names of tables in the given database as an array.

    Returns the names of tables in the given database as an array.

    Definition Classes
    SQLContext
  99. def tableNames(): Array[String]

    Returns the names of tables in the current database as an array.

    Returns the names of tables in the current database as an array.

    Definition Classes
    SQLContext
  100. def tables(databaseName: String): DataFrame

    Returns a DataFrame containing names of existing tables in the given database.

    Returns a DataFrame containing names of existing tables in the given database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
  101. def tables(): DataFrame

    Returns a DataFrame containing names of existing tables in the current database.

    Returns a DataFrame containing names of existing tables in the current database. The returned DataFrame has two columns, tableName and isTemporary (a Boolean indicating if a table is a temporary one or not).

    Definition Classes
    SQLContext
  102. def toString(): String

    Definition Classes
    AnyRef → Any
  103. val udf: UDFRegistration

    A collection of methods for registering user-defined functions (UDF).

    A collection of methods for registering user-defined functions (UDF).

    The following example registers a Scala closure as UDF:

    sqlContext.udf.register("myUdf", (arg1: Int, arg2: String) => arg2 + arg1)

    The following example registers a UDF in Java:

    sqlContext.udf().register("myUDF",
     new UDF2<Integer, String, String>() {
         @Override
         public String call(Integer arg1, String arg2) {
             return arg2 + arg1;
         }
    }, DataTypes.StringType);

    Or, to use Java 8 lambda syntax:

    sqlContext.udf().register("myUDF",
    (Integer arg1, String arg2) -> arg2 + arg1),
    DataTypes.StringType);
    Definition Classes
    SQLContext
  104. def uncacheTable(tableName: String): Unit

    Removes the specified table from the in-memory cache.

    Removes the specified table from the in-memory cache.

    Definition Classes
    SQLContext
  105. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  106. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  107. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def applySchema(rdd: JavaRDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use createDataFrame

  2. def applySchema(rdd: RDD[_], beanClass: Class[_]): DataFrame

    Applies a schema to an RDD of Java Beans.

    Applies a schema to an RDD of Java Beans.

    WARNING: Since there is no guaranteed ordering for fields in a Java Bean, SELECT * queries will return the columns in an undefined order.

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use createDataFrame

  3. def applySchema(rowRDD: JavaRDD[Row], schema: StructType): DataFrame

    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use createDataFrame

  4. def applySchema(rowRDD: RDD[Row], schema: StructType): DataFrame

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows by applying a schema to this RDD.

    :: DeveloperApi :: Creates a DataFrame from an RDD containing Rows by applying a schema to this RDD. It is important to make sure that the structure of every Row of the provided RDD matches the provided schema. Otherwise, there will be runtime exception. Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    
    val schema =
      StructType(
        StructField("name", StringType, false) ::
        StructField("age", IntegerType, true) :: Nil)
    
    val people =
      sc.textFile("examples/src/main/resources/people.txt").map(
        _.split(",")).map(p => Row(p(0), p(1).trim.toInt))
    val dataFrame = sqlContext. applySchema(people, schema)
    dataFrame.printSchema
    // root
    // |-- name: string (nullable = false)
    // |-- age: integer (nullable = true)
    
    dataFrame.registerTempTable("people")
    sqlContext.sql("select name from people").collect.foreach(println)
    Definition Classes
    SQLContext
    Annotations
    @deprecated
    Deprecated

    (Since version 1.3.0) use createDataFrame

Inherited from SQLContext

Inherited from Serializable

Inherited from Serializable

Inherited from Logging

Inherited from AnyRef

Inherited from Any

Basic Operations

Cached Table Management

Configuration

Custom DataFrame Creation

Persistent Catalog DDL

Generic Data Sources

Specific Data Sources

Ungrouped