Nook001 commented on PR #26941: URL: https://github.com/apache/datafusion/pull/26941#issuecomment-6536143870
Quick update: I'm testing a further refinement to improve memory accounting accuracy for small primitive `COUNT(DISTINCT)` tables. The candidate replaces `std::collections::HashSet` with DataFusion's existing `datafusion_common::HashSet` (hashbrown), retaining the explicit `RandomState`. This allows us to use `allocation_size()` directly, eliminating the remaining 16-byte estimation residual without hard-coded corrections or capacity-based layout assumptions. Preliminary Windows/MSVC validation is promising: - 41 scenarios across 5 accumulator types (Float64, Int32, Int64, Decimal128, Decimal256). - All 10,250 candidate retained-heap measurements showed zero estimation error. - Targeted unit tests, `aggregate.slt`, formatting, and Clippy passed. The candidate has not yet been committed or pushed. Windows performance measurements were affected by concurrent background workloads, so I am not treating those results as representative. I'm proceeding with Linux allocation and performance validation before incorporating this refinement into the PR. I'll update the implementation and testing summary once the results are confirmed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
