pyspark.sql.functions.variant_get#

pyspark.sql.functions.variant_get(v, path, targetType)[source]#

Extracts a sub-variant from v according to path, and then cast the sub-variant to targetType. Returns null if the path does not exist. Throws an exception if the cast fails.

New in version 4.0.0.

Parameters

vColumn or str: a variant column or column name
pathColumn or str: a column containing the extraction path strings or a string representing the extraction path. A valid path should start with $ and is followed by zero or more segments like [123], .name, [‘name’], or [“name”].
targetTypestr: the target data type to cast into, in a DDL-formatted string

Returns

Column: a column of targetType representing the extracted result

Examples

>>> df = spark.createDataFrame([ {'json': '''{ "a" : 1 }''', 'path': '$.a'} ])
>>> df.select(variant_get(parse_json(df.json), "$.a", "int").alias("r")).collect()
[Row(r=1)]
>>> df.select(variant_get(parse_json(df.json), "$.b", "int").alias("r")).collect()
[Row(r=None)]
>>> df.select(variant_get(parse_json(df.json), df.path, "int").alias("r")).collect()
[Row(r=1)]