pyspark.sql.functions.tuple_sketch_agg_double#

pyspark.sql.functions.tuple_sketch_agg_double(key, summary, lgNomEntries=None, mode=None)[source]#

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with double summaries built from the key and summary columns.

Added in version 4.2.0.

Parameters:

keyColumn or column name: The column containing key values. A column that evaluates to an array, binary, double, float, integer, long, or string.
summaryColumn or column name: The column containing double summary values. A column that evaluates to a double.
lgNomEntriesColumn or int, optional: The log-base-2 of nominal entries (must be between 4 and 26, defaults to 12) A column that evaluates to an integer.
modeColumn or str, optional: The summary mode: “sum” (default), “min”, “max”, or “alwaysone”

Returns:

Column: The binary representation of the TupleSketch.

See also

pyspark.sql.functions.tuple_sketch_estimate_double()
pyspark.sql.functions.tuple_sketch_summary_double()
pyspark.sql.functions.tuple_union_agg_double()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1, 10.0), (2, 20.0), (2, 30.0)], ["key", "value"])
>>> df.agg(sf.tuple_sketch_estimate_double(
...     sf.tuple_sketch_agg_double("key", "value"))).show()
+--------------------------------------------------------------------------+
|tuple_sketch_estimate_double(tuple_sketch_agg_double(key, value, 12, sum))|
+--------------------------------------------------------------------------+
|                                                                       2.0|
+--------------------------------------------------------------------------+