unikdahal opened a new issue, #6608: URL: https://github.com/apache/datafusion-comet/issues/6608
## Problem Native `MergeRowsExec` uses `HashSet<i64>` for cardinality validation, while Spark uses `Roaring64Bitmap`. As measured in the [#5318 review](https://github.com/apache/datafusion-comet/pull/5318#discussion_r4178476396), for **5M row IDs**: | Representation | Memory | | --- | ---: | | `HashSet<i64>` | ~75 MB | | Roaring, dense | ~0.6 MB | | Roaring, shuffled | ~5–17 MB | Since this state cannot spill, native MERGE can hit `ResourcesExhausted` where Spark still fits in memory. ## Proposed solution Replace `HashSet<i64>` with `RoaringTreemap` while preserving duplicate detection, existing cardinality semantics, and DataFusion memory accounting. Add coverage for dense/shuffled IDs, cross-batch duplicates, and bounded-memory cases. Related: #5318, #5122 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
