public abstract class UserDefinedType<UserType> extends DataType implements scala.Serializable
This interface allows a user to make their own classes more interoperable with SparkSQL;
e.g., by creating a UserDefinedType
for a class X, it becomes possible to create
a DataFrame
which has class X in the schema.
For SparkSQL to recognize UDTs, the UDT must be annotated with
SQLUserDefinedType
.
The conversion via serialize
occurs when instantiating a DataFrame
from another RDD.
The conversion via deserialize
occurs when reading from a DataFrame
.
Constructor and Description |
---|
UserDefinedType() |
Modifier and Type | Method and Description |
---|---|
UserDefinedType<UserType> |
asNullable()
For UDT, asNullable will not change the nullability of its internal sqlType and just returns
itself.
|
int |
defaultSize()
The default size of a value of the UserDefinedType is 4096 bytes.
|
abstract UserType |
deserialize(Object datum)
Convert a SQL datum to the user type
|
org.json4s.JsonAST.JValue |
jsonValue() |
String |
pyUDT()
Paired Python UDT class, if exists.
|
abstract Object |
serialize(Object obj)
Convert the user type to a SQL datum
|
abstract DataType |
sqlType()
Underlying storage type for this UDT
|
abstract Class<UserType> |
userClass()
Class object for the UserType
|
equalsIgnoreCompatibleNullability, equalsIgnoreNullability, fromCaseClassString, fromJson, isPrimitive, json, prettyJson, sameType, simpleString, typeName, unapply
public abstract DataType sqlType()
public String pyUDT()
public abstract Object serialize(Object obj)
TODO: Can we make this take obj: UserType? The issue is in ScalaReflection.convertToCatalyst, where we need to convert Any to UserType.
public abstract UserType deserialize(Object datum)
public abstract Class<UserType> userClass()
public int defaultSize()
defaultSize
in class DataType
public UserDefinedType<UserType> asNullable()
asNullable
in class DataType