org.apache.spark.sql.types

Class DataType

Object

org.apache.spark.sql.types.DataType

Direct Known Subclasses:

ArrayType, BinaryType, BooleanType, CalendarIntervalType, CharType, DateType, DayTimeIntervalType, MapType, NullType, NumericType, ObjectType, StringType, StructType, TimestampNTZType, TimestampType, UserDefinedType, VarcharType, YearMonthIntervalType

public abstract class DataType
extends Object

The base type of all Spark SQL data types.

Since:

1.3.0

Constructor Summary

Constructors

Constructor and Description
DataType()

Method Summary

Modifier and Type	Method and Description
String	catalogString() String representation for the type saved in external catalogs.
abstract int	defaultSize() The default size of a value of this data type, used internally for size estimation.
static boolean	equalsIgnoreCaseAndNullability(DataType from, DataType to) Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.
static boolean	equalsIgnoreNullability(DataType left, DataType right) Compares two types, ignoring nullability of ArrayType, MapType, StructType.
static boolean	equalsStructurally(DataType from, DataType to, boolean ignoreNullability) Returns true if the two data types share the same "shape", i.e.
static boolean	equalsStructurallyByName(DataType from, DataType to, scala.Function2<String,String,Object> resolver) Returns true if the two data types have the same field names in order recursively.
static DataType	fromDDL(String ddl)
static DataType	fromJson(String json)
String	json() The compact JSON representation of this data type.
static DataType	parseTypeWithFallback(String schema, scala.Function1<String,DataType> parser, scala.Function1<String,DataType> fallbackParser) Parses data type from a string with schema.
String	prettyJson() The pretty (i.e.
String	simpleString() Readable string representation for the type.
String	sql()
String	typeName() Name of the type used in JSON serialization.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataType

public DataType()

Method Detail

fromDDL

public static DataType fromDDL(String ddl)

parseTypeWithFallback

public static DataType parseTypeWithFallback(String schema,
                                             scala.Function1<String,DataType> parser,
                                             scala.Function1<String,DataType> fallbackParser)

Parses data type from a string with schema. It calls parser for schema. If it fails, calls fallbackParser. If the fallback function fails too, combines error message from parser and fallbackParser.

Parameters:

schema - The schema string to parse by parser or fallbackParser.

parser - The function that should be invoke firstly.

fallbackParser - The function that is called when parser fails.

Returns:

The data type parsed from the schema schema.

fromJson

public static DataType fromJson(String json)

equalsStructurally

public static boolean equalsStructurally(DataType from,
                                         DataType to,
                                         boolean ignoreNullability)

Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.

Parameters:

ignoreNullability - whether to ignore nullability when comparing the types

from - (undocumented)

to - (undocumented)

Returns:

(undocumented)

equalsStructurallyByName

public static boolean equalsStructurallyByName(DataType from,
                                               DataType to,
                                               scala.Function2<String,String,Object> resolver)

Returns true if the two data types have the same field names in order recursively.

Parameters:

from - (undocumented)

to - (undocumented)

resolver - (undocumented)

Returns:

(undocumented)

equalsIgnoreNullability

public static boolean equalsIgnoreNullability(DataType left,
                                              DataType right)

Compares two types, ignoring nullability of ArrayType, MapType, StructType.

Parameters:

left - (undocumented)

right - (undocumented)

Returns:

(undocumented)

equalsIgnoreCaseAndNullability

public static boolean equalsIgnoreCaseAndNullability(DataType from,
                                                     DataType to)

Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.

Parameters:

from - (undocumented)

to - (undocumented)

Returns:

(undocumented)

defaultSize

public abstract int defaultSize()

The default size of a value of this data type, used internally for size estimation.

Returns:

(undocumented)

typeName

public String typeName()

Name of the type used in JSON serialization.

json

public String json()

The compact JSON representation of this data type.

prettyJson

public String prettyJson()

The pretty (i.e. indented) JSON representation of this data type.

simpleString

public String simpleString()

Readable string representation for the type.

catalogString

public String catalogString()

String representation for the type saved in external catalogs.

sql

public String sql()