Class/Object

org.apache.spark.sql.hive.execution

HiveScriptIOSchema

Related Docs: object HiveScriptIOSchema | package execution

Permalink

case class HiveScriptIOSchema(inputRowFormat: Seq[(String, String)], outputRowFormat: Seq[(String, String)], inputSerdeClass: Option[String], outputSerdeClass: Option[String], inputSerdeProps: Seq[(String, String)], outputSerdeProps: Seq[(String, String)], recordReaderClass: Option[String], recordWriterClass: Option[String], schemaLess: Boolean) extends HiveInspectors with Product with Serializable

The wrapper class of Hive input and output schema properties

Source
ScriptTransformation.scala
Linear Supertypes
Serializable, Serializable, Product, Equals, HiveInspectors, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. HiveScriptIOSchema
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. HiveInspectors
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new HiveScriptIOSchema(inputRowFormat: Seq[(String, String)], outputRowFormat: Seq[(String, String)], inputSerdeClass: Option[String], outputSerdeClass: Option[String], inputSerdeProps: Seq[(String, String)], outputSerdeProps: Seq[(String, String)], recordReaderClass: Option[String], recordWriterClass: Option[String], schemaLess: Boolean)

    Permalink

Type Members

  1. implicit class typeInfoConversions extends AnyRef

    Permalink
    Definition Classes
    HiveInspectors

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  5. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  6. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  7. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  8. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  9. def initInputSerDe(input: Seq[Expression]): Option[(AbstractSerDe, ObjectInspector)]

    Permalink
  10. def initOutputSerDe(output: Seq[Attribute]): Option[(AbstractSerDe, StructObjectInspector)]

    Permalink
  11. val inputRowFormat: Seq[(String, String)]

    Permalink
  12. val inputRowFormatMap: Map[String, String]

    Permalink
  13. val inputSerdeClass: Option[String]

    Permalink
  14. val inputSerdeProps: Seq[(String, String)]

    Permalink
  15. def inspectorToDataType(inspector: ObjectInspector): DataType

    Permalink
    Definition Classes
    HiveInspectors
  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  17. def javaClassToDataType(clz: Class[_]): DataType

    Permalink
    Definition Classes
    HiveInspectors
  18. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  19. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  20. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  21. val outputRowFormat: Seq[(String, String)]

    Permalink
  22. val outputRowFormatMap: Map[String, String]

    Permalink
  23. val outputSerdeClass: Option[String]

    Permalink
  24. val outputSerdeProps: Seq[(String, String)]

    Permalink
  25. def recordReader(inputStream: InputStream, conf: Configuration): Option[RecordReader]

    Permalink
  26. val recordReaderClass: Option[String]

    Permalink
  27. def recordWriter(outputStream: OutputStream, conf: Configuration): Option[RecordWriter]

    Permalink
  28. val recordWriterClass: Option[String]

    Permalink
  29. val schemaLess: Boolean

    Permalink
  30. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  31. def toInspector(expr: Expression): ObjectInspector

    Permalink

    Map the catalyst expression to ObjectInspector, however, if the expression is Literal or foldable, a constant writable object inspector returns; Otherwise, we always get the object inspector according to its data type(in catalyst)

    Map the catalyst expression to ObjectInspector, however, if the expression is Literal or foldable, a constant writable object inspector returns; Otherwise, we always get the object inspector according to its data type(in catalyst)

    expr

    Catalyst expression to be mapped

    returns

    Hive java objectinspector (recursively).

    Definition Classes
    HiveInspectors
  32. def toInspector(dataType: DataType): ObjectInspector

    Permalink

    dataType

    Catalyst data type

    returns

    Hive java object inspector (recursively), not the Writable ObjectInspector We can easily map to the Hive built-in object inspector according to the data type.

    Definition Classes
    HiveInspectors
  33. def unwrap(data: Any, oi: ObjectInspector): Any

    Permalink

    Converts hive types to native catalyst types.

    Converts hive types to native catalyst types.

    data

    the data in Hive type

    oi

    the ObjectInspector associated with the Hive Type

    returns

    convert the data into catalyst type TODO return the function of (data => Any) instead for performance consideration Strictly follows the following order in unwrapping (constant OI has the higher priority): Constant Null object inspector => return null Constant object inspector => extract the value from constant object inspector Check whether the data is null => return null if true If object inspector prefers writable => extract writable from data and then get the catalyst type from the writable Extract the java object directly from the object inspector NOTICE: the complex data type requires recursive unwrapping.

    Definition Classes
    HiveInspectors
  34. def unwrapperFor(field: StructField): (Any, MutableRow, Int) ⇒ Unit

    Permalink

    Builds specific unwrappers ahead of time according to object inspector types to avoid pattern matching and branching costs per row.

    Builds specific unwrappers ahead of time according to object inspector types to avoid pattern matching and branching costs per row.

    Definition Classes
    HiveInspectors
  35. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  36. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  37. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  38. def wrap(row: Seq[Any], inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

    Permalink
    Definition Classes
    HiveInspectors
  39. def wrap(row: InternalRow, inspectors: Seq[ObjectInspector], cache: Array[AnyRef], dataTypes: Array[DataType]): Array[AnyRef]

    Permalink
    Definition Classes
    HiveInspectors
  40. def wrap(a: Any, oi: ObjectInspector, dataType: DataType): AnyRef

    Permalink

    Converts native catalyst types to the types expected by Hive

    Converts native catalyst types to the types expected by Hive

    a

    the value to be wrapped

    oi

    This ObjectInspector associated with the value returned by this function, and the ObjectInspector should also be consistent with those returned from toInspector: DataType => ObjectInspector and toInspector: Expression => ObjectInspector Strictly follows the following order in wrapping (constant OI has the higher priority): Constant object inspector => return the bundled value of Constant object inspector Check whether the a is null => return null if true If object inspector prefers writable object => return a Writable for the given data a Map the catalyst data to the boxed java primitive NOTICE: the complex data type requires recursive wrapping.

    Definition Classes
    HiveInspectors
  41. def wrapperFor(oi: ObjectInspector, dataType: DataType): (Any) ⇒ Any

    Permalink

    Wraps with Hive types based on object inspector.

    Wraps with Hive types based on object inspector. TODO: Consolidate all hive OI/data interface code.

    Attributes
    protected
    Definition Classes
    HiveInspectors

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from HiveInspectors

Inherited from AnyRef

Inherited from Any

Ungrouped