pyspark.sql.functions.array_compact¶

pyspark.sql.functions.array_compact(col: ColumnOrName) → pyspark.sql.column.Column[source]¶

Collection function: removes null values from the array.

New in version 3.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colColumn or str: name of column or expression

Returns

Column: an array by exluding the null values.

Examples

>>> df = spark.createDataFrame([([1, None, 2, 3],), ([4, 5, None, 4],)], ['data'])
>>> df.select(array_compact(df.data)).collect()
[Row(array_compact(data)=[1, 2, 3]), Row(array_compact(data)=[4, 5, 4])]

pyspark.sql.functions.array_except pyspark.sql.functions.transform