Packages

package types

Contains a type system for attributes produced by relations, including complex types like structs, arrays and maps.

Source
package.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. types
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class ArrayType(elementType: DataType, containsNull: Boolean) extends DataType with Product with Serializable

    The data type for collections of multiple values.

    The data type for collections of multiple values. Internally these are represented as columns that contain a scala.collection.Seq.

    Please use DataTypes.createArrayType() to create a specific instance.

    An ArrayType object comprises two fields, elementType: DataType and containsNull: Boolean. The field of elementType is used to specify the type of array elements. The field of containsNull is used to specify if the array can have null values.

    elementType

    The data type of values.

    containsNull

    Indicates if the array can have null values

    Annotations
    @Stable()
    Since

    1.3.0

  2. abstract class AtomicType extends DataType

    An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.

    An internal type used to represent everything that is not null, UDTs, arrays, structs, and maps.

    Attributes
    protected[sql]
  3. class BinaryType extends AtomicType

    The data type representing Array[Byte] values.

    The data type representing Array[Byte] values. Please use the singleton DataTypes.BinaryType.

    Annotations
    @Stable()
  4. class BooleanType extends AtomicType

    The data type representing Boolean values.

    The data type representing Boolean values. Please use the singleton DataTypes.BooleanType.

    Annotations
    @Stable()
    Since

    1.3.0

  5. class ByteType extends IntegralType

    The data type representing Byte values.

    The data type representing Byte values. Please use the singleton DataTypes.ByteType.

    Annotations
    @Stable()
    Since

    1.3.0

  6. class CalendarIntervalType extends DataType

    The data type representing calendar intervals.

    The data type representing calendar intervals. The calendar interval is stored internally in three components: an integer value representing the number of months in this interval, an integer value representing the number of days in this interval, a long value representing the number of microseconds in this interval.

    Please use the singleton DataTypes.CalendarIntervalType to refer the type.

    Annotations
    @Stable()
    Since

    1.5.0

    Note

    Calendar intervals are not comparable.

  7. case class CharType(length: Int) extends AtomicType with Product with Serializable
    Annotations
    @Experimental()
  8. abstract class DataType extends AbstractDataType

    The base type of all Spark SQL data types.

    The base type of all Spark SQL data types.

    Annotations
    @Stable() @JsonSerialize() @JsonDeserialize()
    Since

    1.3.0

  9. class DataTypes extends AnyRef
    Annotations
    @Stable()
  10. class DateType extends DatetimeType

    The date type represents a valid date in the proleptic Gregorian calendar.

    The date type represents a valid date in the proleptic Gregorian calendar. Valid range is [0001-01-01, 9999-12-31].

    Please use the singleton DataTypes.DateType to refer the type.

    Annotations
    @Stable()
    Since

    1.3.0

  11. case class DayTimeIntervalType(startField: Byte, endField: Byte) extends AnsiIntervalType with Product with Serializable

    The type represents day-time intervals of the SQL standard.

    The type represents day-time intervals of the SQL standard. A day-time interval is made up of a contiguous subset of the following fields:

    • SECOND, seconds within minutes and possibly fractions of a second [0..59.999999],
    • MINUTE, minutes within hours [0..59],
    • HOUR, hours within days [0..23],
    • DAY, days in the range [0..106751991].

    DayTimeIntervalType represents positive as well as negative day-time intervals.

    startField

    The leftmost field which the type comprises of. Valid values: 0 (DAY), 1 (HOUR), 2 (MINUTE), 3 (SECOND).

    endField

    The rightmost field which the type comprises of. Valid values: 0 (DAY), 1 (HOUR), 2 (MINUTE), 3 (SECOND).

    Annotations
    @Unstable()
    Since

    3.2.0

  12. final class Decimal extends Ordered[Decimal] with Serializable

    A mutable implementation of BigDecimal that can hold a Long if values are small enough.

    A mutable implementation of BigDecimal that can hold a Long if values are small enough.

    The semantics of the fields are as follows: - _precision and _scale represent the SQL precision and scale we are looking for - If decimalVal is set, it represents the whole decimal value - Otherwise, the decimal value is longVal / (10 ** _scale)

    Note, for values between -1.0 and 1.0, precision digits are only counted after dot.

    Annotations
    @Unstable()
  13. case class DecimalType(precision: Int, scale: Int) extends FractionalType with Product with Serializable

    The data type representing java.math.BigDecimal values.

    The data type representing java.math.BigDecimal values. A Decimal that must have fixed precision (the maximum number of digits) and scale (the number of digits on right side of dot).

    The precision can be up to 38, scale can also be up to 38 (less or equal to precision).

    The default precision and scale is (10, 0).

    Please use DataTypes.createDecimalType() to create a specific instance.

    Annotations
    @Stable()
    Since

    1.3.0

  14. class DoubleType extends FractionalType

    The data type representing Double values.

    The data type representing Double values. Please use the singleton DataTypes.DoubleType.

    Annotations
    @Stable()
    Since

    1.3.0

  15. class FloatType extends FractionalType

    The data type representing Float values.

    The data type representing Float values. Please use the singleton DataTypes.FloatType.

    Annotations
    @Stable()
    Since

    1.3.0

  16. class IntegerType extends IntegralType

    The data type representing Int values.

    The data type representing Int values. Please use the singleton DataTypes.IntegerType.

    Annotations
    @Stable()
    Since

    1.3.0

  17. class LongType extends IntegralType

    The data type representing Long values.

    The data type representing Long values. Please use the singleton DataTypes.LongType.

    Annotations
    @Stable()
    Since

    1.3.0

  18. case class MapType(keyType: DataType, valueType: DataType, valueContainsNull: Boolean) extends DataType with Product with Serializable

    The data type for Maps.

    The data type for Maps. Keys in a map are not allowed to have null values.

    Please use DataTypes.createMapType() to create a specific instance.

    keyType

    The data type of map keys.

    valueType

    The data type of map values.

    valueContainsNull

    Indicates if map values have null values.

    Annotations
    @Stable()
  19. sealed class Metadata extends Serializable

    Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata].

    Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata]. JSON is used for serialization.

    The default constructor is private. User should use either MetadataBuilder or Metadata.fromJson() to create Metadata instances.

    Annotations
    @Stable()
    Since

    1.3.0

  20. class MetadataBuilder extends AnyRef

    Builder for Metadata.

    Builder for Metadata. If there is a key collision, the latter will overwrite the former.

    Annotations
    @Stable()
    Since

    1.3.0

  21. class NullType extends DataType

    The data type representing NULL values.

    The data type representing NULL values. Please use the singleton DataTypes.NullType.

    Annotations
    @Stable()
    Since

    1.3.0

  22. abstract class NumericType extends AtomicType

    Numeric data types.

    Numeric data types.

    Annotations
    @Stable()
    Since

    1.3.0

  23. case class ObjectType(cls: Class[_]) extends DataType with Product with Serializable

    Represents a JVM object that is passing through Spark SQL expression evaluation.

  24. class SQLUserDefinedType extends Annotation with Annotation with ClassfileAnnotation
    Annotations
    @DeveloperApi() @Retention() @Target()
  25. class ShortType extends IntegralType

    The data type representing Short values.

    The data type representing Short values. Please use the singleton DataTypes.ShortType.

    Annotations
    @Stable()
    Since

    1.3.0

  26. class StringType extends AtomicType

    The data type representing String values.

    The data type representing String values. Please use the singleton DataTypes.StringType.

    Annotations
    @Stable()
    Since

    1.3.0

  27. case class StructField(name: String, dataType: DataType, nullable: Boolean = true, metadata: Metadata = Metadata.empty) extends Product with Serializable

    A field inside a StructType.

    A field inside a StructType.

    name

    The name of this field.

    dataType

    The data type of this field.

    nullable

    Indicates if values of this field can be null values.

    metadata

    The metadata of this field. The metadata should be preserved during transformation if the content of the column is not modified, e.g, in selection.

    Annotations
    @Stable()
    Since

    1.3.0

  28. case class StructType(fields: Array[StructField]) extends DataType with Seq[StructField] with Product with Serializable

    A StructType object can be constructed by

    A StructType object can be constructed by

    StructType(fields: Seq[StructField])

    For a StructType object, one or multiple StructFields can be extracted by names. If multiple StructFields are extracted, a StructType object will be returned. If a provided name does not have a matching field, it will be ignored. For the case of extracting a single StructField, a null will be returned.

    Scala Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    
    val struct =
      StructType(
        StructField("a", IntegerType, true) ::
        StructField("b", LongType, false) ::
        StructField("c", BooleanType, false) :: Nil)
    
    // Extract a single StructField.
    val singleField = struct("b")
    // singleField: StructField = StructField(b,LongType,false)
    
    // If this struct does not have a field called "d", it throws an exception.
    struct("d")
    // java.lang.IllegalArgumentException: d does not exist.
    //   ...
    
    // Extract multiple StructFields. Field names are provided in a set.
    // A StructType object will be returned.
    val twoFields = struct(Set("b", "c"))
    // twoFields: StructType =
    //   StructType(StructField(b,LongType,false), StructField(c,BooleanType,false))
    
    // Any names without matching fields will throw an exception.
    // For the case shown below, an exception is thrown due to "d".
    struct(Set("b", "c", "d"))
    // java.lang.IllegalArgumentException: d does not exist.
    //    ...

    A org.apache.spark.sql.Row object is used as a value of the StructType.

    Scala Example:

    import org.apache.spark.sql._
    import org.apache.spark.sql.types._
    
    val innerStruct =
      StructType(
        StructField("f1", IntegerType, true) ::
        StructField("f2", LongType, false) ::
        StructField("f3", BooleanType, false) :: Nil)
    
    val struct = StructType(
      StructField("a", innerStruct, true) :: Nil)
    
    // Create a Row with the schema defined by struct
    val row = Row(Row(1, 2, true))
    Annotations
    @Stable()
    Since

    1.3.0

  29. class TimestampNTZType extends DatetimeType

    The timestamp without time zone type represents a local time in microsecond precision, which is independent of time zone.

    The timestamp without time zone type represents a local time in microsecond precision, which is independent of time zone. Its valid range is [0001-01-01T00:00:00.000000, 9999-12-31T23:59:59.999999]. To represent an absolute point in time, use TimestampType instead.

    Please use the singleton DataTypes.TimestampNTZType to refer the type.

    Annotations
    @Unstable()
    Since

    3.4.0

  30. class TimestampType extends DatetimeType

    The timestamp type represents a time instant in microsecond precision.

    The timestamp type represents a time instant in microsecond precision. Valid range is [0001-01-01T00:00:00.000000Z, 9999-12-31T23:59:59.999999Z] where the left/right-bound is a date and time of the proleptic Gregorian calendar in UTC+00:00.

    Please use the singleton DataTypes.TimestampType to refer the type.

    Annotations
    @Stable()
    Since

    1.3.0

  31. abstract class UserDefinedType[UserType >: Null] extends DataType with Serializable

    The data type for User Defined Types (UDTs).

    The data type for User Defined Types (UDTs).

    This interface allows a user to make their own classes more interoperable with SparkSQL; e.g., by creating a UserDefinedType for a class X, it becomes possible to create a DataFrame which has class X in the schema.

    For SparkSQL to recognize UDTs, the UDT must be annotated with SQLUserDefinedType.

    The conversion via serialize occurs when instantiating a DataFrame from another RDD. The conversion via deserialize occurs when reading from a DataFrame.

    Annotations
    @DeveloperApi() @Since( "3.2.0" )
  32. case class VarcharType(length: Int) extends AtomicType with Product with Serializable
    Annotations
    @Experimental()
  33. case class YearMonthIntervalType(startField: Byte, endField: Byte) extends AnsiIntervalType with Product with Serializable

    The type represents year-month intervals of the SQL standard.

    The type represents year-month intervals of the SQL standard. A year-month interval is made up of a contiguous subset of the following fields:

    • MONTH, months within years [0..11],
    • YEAR, years in the range [0..178956970].

    YearMonthIntervalType represents positive as well as negative year-month intervals.

    startField

    The leftmost field which the type comprises of. Valid values: 0 (YEAR), 1 (MONTH).

    endField

    The rightmost field which the type comprises of. Valid values: 0 (YEAR), 1 (MONTH).

    Annotations
    @Unstable()
    Since

    3.2.0

Value Members

  1. object AnyDataType extends AbstractDataType with Serializable

    An AbstractDataType that matches any concrete data types.

    An AbstractDataType that matches any concrete data types.

    Attributes
    protected[sql]
  2. object ArrayType extends AbstractDataType with Serializable

    Companion object for ArrayType.

    Companion object for ArrayType.

    Annotations
    @Stable()
    Since

    1.3.0

  3. object AtomicType
  4. object BinaryType extends BinaryType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  5. object BooleanType extends BooleanType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  6. object ByteType extends ByteType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  7. object CalendarIntervalType extends CalendarIntervalType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.5.0

  8. object DataType

    Annotations
    @Stable()
    Since

    1.3.0

  9. object DateType extends DateType with Product with Serializable

    The companion case object and the DateType class is separated so the companion object also subclasses the class.

    The companion case object and the DateType class is separated so the companion object also subclasses the class. Otherwise, the companion object would be of type "DateType$" in byte code. The DateType class is defined with a private constructor so its companion object is the only possible instantiation.

    Annotations
    @Stable()
    Since

    1.3.0

  10. object DayTimeIntervalType extends AbstractDataType with Product with Serializable

    Extra factory methods and pattern matchers for DayTimeIntervalType.

    Extra factory methods and pattern matchers for DayTimeIntervalType.

    Annotations
    @Unstable()
    Since

    3.2.0

  11. object Decimal extends Serializable
    Annotations
    @Unstable()
  12. object DecimalType extends AbstractDataType with Serializable

    Extra factory methods and pattern matchers for Decimals.

    Extra factory methods and pattern matchers for Decimals.

    Annotations
    @Stable()
    Since

    1.3.0

  13. object DoubleType extends DoubleType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  14. object FloatType extends FloatType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  15. object IntegerType extends IntegerType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  16. object LongType extends LongType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  17. object MapType extends AbstractDataType with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  18. object Metadata extends Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  19. object NullType extends NullType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  20. object ObjectType extends AbstractDataType with Serializable
  21. object ShortType extends ShortType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  22. object StringType extends StringType with Product with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  23. object StructType extends AbstractDataType with Serializable

    Annotations
    @Stable()
    Since

    1.3.0

  24. object TimestampNTZType extends TimestampNTZType with Product with Serializable

    The companion case object and its class is separated so the companion object also subclasses the TimestampNTZType class.

    The companion case object and its class is separated so the companion object also subclasses the TimestampNTZType class. Otherwise, the companion object would be of type "TimestampNTZType" in byte code. Defined with a private constructor so the companion object is the only possible instantiation.

    Annotations
    @Unstable()
    Since

    3.4.0

  25. object TimestampType extends TimestampType with Product with Serializable

    The companion case object and its class is separated so the companion object also subclasses the TimestampType class.

    The companion case object and its class is separated so the companion object also subclasses the TimestampType class. Otherwise, the companion object would be of type "TimestampType$" in byte code. Defined with a private constructor so the companion object is the only possible instantiation.

    Annotations
    @Stable()
    Since

    1.3.0

  26. object UDTRegistration extends Serializable with Logging

    This object keeps the mappings between user classes and their User Defined Types (UDTs).

    This object keeps the mappings between user classes and their User Defined Types (UDTs). Previously we use the annotation SQLUserDefinedType to register UDTs for user classes. However, by doing this, we add SparkSQL dependency on user classes. This object provides alternative approach to register UDTs for user classes.

    Annotations
    @DeveloperApi() @Since( "3.2.0" )
  27. object YearMonthIntervalType extends AbstractDataType with Product with Serializable

    Extra factory methods and pattern matchers for YearMonthIntervalType.

    Extra factory methods and pattern matchers for YearMonthIntervalType.

    Annotations
    @Unstable()
    Since

    3.2.0

Inherited from AnyRef

Inherited from Any

Ungrouped