andygrove opened a new issue, #4984: URL: https://github.com/apache/datafusion-comet/issues/4984
## Is your feature request related to a problem or challenge? PR #4819 added a native Spark-compatible `approx_count_distinct` (a faithful port of Spark's `HyperLogLogPlusPlus` / `HyperLogLogPlusPlusHelper`) as a Comet-only aggregate in `native/spark-expr/src/agg_funcs/hll_plus_plus.rs`, along with the bias-correction tables in `hll_plus_plus_const.rs`. This is a natural fit for the upstream `datafusion-spark` crate (`datafusion/datafusion/spark/src/function/aggregate/`, which already carries `avg`, `try_sum`, `collect`), so the algorithm and the Spark bias-correction tables could be shared rather than living only in Comet. DataFusion's own `approx_distinct` cannot be reused: it implements the 2017 Ertl variant (closed-form estimator, no lookup tables, fixed precision `HLL_P = 14`, one `u8` per register, foldhash seed 0), whereas Spark uses the 2013 Google HLL++ variant (xxhash64 seed 42, `p` derived from `relativeSD`, 6-bit registers packed 10 per long, empirical bias-correction plus linear-counting tables). Every compatibility-critical axis differs. ## Describe the solution you'd like Move the HLL++ algorithm and bias-correction tables into the `datafusion-spark` crate so they can be shared. The main coupling to unwind is the hashing: the Comet implementation reuses Comet's `create_xxhash64_hashes` (Spark's `XxHash64`, seed 42, with `NormalizeNaNAndZero` applied to floats first). Upstreaming needs a Spark-compatible xxhash64 available in `datafusion-spark`, or the port parameterized over the hash function. ## Additional context Follow-up to PR #4819 (part of #4098). Landing in Comet first is the expected path; this issue tracks the eventual upstreaming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
