MapType

class pyspark.sql.types.MapType(keyType, valueType, valueContainsNull=True)

Map data type.

Parameters

keyTypeDataType

DataType of the keys in the map.

valueTypeDataType

DataType of the values in the map.

valueContainsNullbool, optional

indicates whether values can contain null (None) values.

Notes

Keys in a map data type are not allowed to be null (None).

Examples

>>> from pyspark.sql.types import IntegerType, FloatType, MapType, StringType

The below example demonstrates how to create class:MapType:

>>> map_type = MapType(StringType(), IntegerType())

The values of the map can contain null (None) values by default:

>>> (MapType(StringType(), IntegerType())
...        == MapType(StringType(), IntegerType(), True))
True
>>> (MapType(StringType(), IntegerType(), False)
...        == MapType(StringType(), FloatType()))
False

Methods

fromDDL(ddl)	Creates DataType for a given DDL-formatted string.
fromInternal(obj)	Converts an internal SQL object into a native Python object.
fromJson(json[, fieldPath, collationsMap])
json()
jsonValue()
needConversion()	Does this type needs conversion between Python object and internal SQL object.
simpleString()
toInternal(obj)	Converts a Python object into an internal SQL object.
toNullable()	Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).
typeName()

Methods Documentation

classmethod fromDDL(ddl)

Creates DataType for a given DDL-formatted string.

New in version 4.0.0.

Parameters

ddlstr

DDL-formatted string representation of types, e.g. pyspark.sql.types.DataType.simpleString, except that top level struct type can omit the struct<> for the compatibility reason with spark.createDataFrame and Python UDFs.

Returns

DataType

Examples

Create a StructType by the corresponding DDL formatted string.

>>> from pyspark.sql.types import DataType
>>> DataType.fromDDL("b string, a int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

Create a single DataType by the corresponding DDL formatted string.

>>> DataType.fromDDL("decimal(10,10)")
DecimalType(10,10)

Create a StructType by the legacy string format.

>>> DataType.fromDDL("b: string, a: int")
StructType([StructField('b', StringType(), True), StructField('a', IntegerType(), True)])

fromInternal(obj)

Converts an internal SQL object into a native Python object.

classmethod fromJson(json, fieldPath='', collationsMap=None)

json()

jsonValue()

needConversion()

Does this type needs conversion between Python object and internal SQL object.

This is used to avoid the unnecessary conversion for ArrayType/MapType/StructType.

simpleString()

toInternal(obj)

Converts a Python object into an internal SQL object.

toNullable()

Returns the same data type but set all nullability fields are true (StructField.nullable, ArrayType.containsNull, and MapType.valueContainsNull).

New in version 4.0.0.

Returns

MapType

Examples

Example 1: Simple nullability conversion

>>> MapType(IntegerType(), StringType(), valueContainsNull=False).toNullable()
MapType(IntegerType(), StringType(), True)

Example 2: Nested nullability conversion

>>> MapType(
...     StringType(),
...     MapType(
...         IntegerType(),
...         ArrayType(IntegerType(), containsNull=False),
...         valueContainsNull=False
...     ),
...     valueContainsNull=False
... ).toNullable()
MapType(StringType(), MapType(IntegerType(), ArrayType(IntegerType(), True), True), True)

classmethod typeName()