Hi all, I’d like to start a discussion about adding support for [Apache DataSketches](https://datasketches.apache.org/) — specifically, Theta and Tuple Sketches — to Spark SQL and DataFrame APIs.
## Motivation These sketches allow scalable approximate set operations (like distinct count, unions, intersections, minus) and are well-suited for large-scale analytics. They are already used in production in systems like Druid, Presto, and others. Integrating them natively into Spark (e.g., as UDAFs or SQL functions) could offer performance and memory efficiency benefits for use cases such as: - Large cardinality distinct counts - Approximate aggregations over streaming/batch data - Set-based operations across datasets ## Proposed Scope - Add Theta and Tuple Sketch-based UDAFs to Spark SQL - Optional integration into `spark.sql` functions (e.g., `approx_count_distinct_sketch`) - Use Apache DataSketches as a dependency (already an incubating Apache project) - Start as an optional module if core integration is too heavy I’m happy to work on a design doc or POC if there’s interest. Thanks, Chris
