pyspark.sql.functions.map_filter

pyspark.sql.functions.map_filter(col: ColumnOrName, f: Callable[[pyspark.sql.column.Column, pyspark.sql.column.Column], pyspark.sql.column.Column]) → pyspark.sql.column.Column

Returns a map whose key-value pairs satisfy a predicate.

New in version 3.1.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colColumn or str

name of column or expression

ffunction

a binary function (k: Column, v: Column) -> Column... Can use methods of Column, functions defined in pyspark.sql.functions and Scala UserDefinedFunctions. Python UserDefinedFunctions are not supported (SPARK-27052).

Returns

Column

filtered map.

Examples

>>> df = spark.createDataFrame([(1, {"foo": 42.0, "bar": 1.0, "baz": 32.0})], ("id", "data"))
>>> row = df.select(map_filter(
...     "data", lambda _, v: v > 30.0).alias("data_filtered")
... ).head()
>>> sorted(row["data_filtered"].items())
[('baz', 32.0), ('foo', 42.0)]

previous

pyspark.sql.functions.transform_values

next

pyspark.sql.functions.map_from_arrays