pyspark.sql.functions.hll_union#

pyspark.sql.functions.hll_union(col1, col2, allowDifferentLgConfigK=None)[source]#

Merges two binary representations of Datasketches HllSketch objects, using a Datasketches Union object. Throws an exception if sketches have different lgConfigK values and allowDifferentLgConfigK is unset or set to false.

Added in version 3.5.0.

Parameters:

col1Column or column name
col2Column or column name
allowDifferentLgConfigKbool, optional: Allow sketches with different lgConfigK values to be merged (defaults to false).

Returns:

Column: The binary representation of the merged HllSketch.

See also

pyspark.sql.functions.hll_union_agg()
pyspark.sql.functions.hll_sketch_agg()
pyspark.sql.functions.hll_sketch_estimate()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1,4),(2,5),(2,5),(3,6)], "struct<v1:int,v2:int>")
>>> df = df.agg(
...     sf.hll_sketch_agg("v1").alias("sketch1"),
...     sf.hll_sketch_agg("v2").alias("sketch2")
... )
>>> df.select(sf.hll_sketch_estimate(sf.hll_union(df.sketch1, "sketch2"))).show()
+-------------------------------------------------------+
|hll_sketch_estimate(hll_union(sketch1, sketch2, false))|
+-------------------------------------------------------+
|                                                      6|
+-------------------------------------------------------+