Nook001 commented on PR #26941: URL: https://github.com/apache/datafusion/pull/26941#issuecomment-6549447660
**Update: Rocky Linux validation and performance investigation** The hashbrown-based memory accounting change is now pushed in `46af5e0a7`. Subsequent Rocky Linux validation confirmed zero retained-heap estimation error across all 10,250 candidate measurements. However, controlled real-SQL benchmarks revealed a reproducible regression in nullable `COUNT(DISTINCT)` execution with the current hashbrown implementation. I have kept this PR in Draft while investigating it. The investigation narrowed the regression to insert-path code generation and inlining. A separate, local hashbrown prototype changes one attribute on `HashMap::insert` from `inline` to `inline(always)` under the existing `inline-more` feature. On Rocky Linux, compared with the matching unmodified hashbrown baseline: - Nullable Int64 SQL: **18.83% faster**. - Decimal report SQL: **15.07% faster**. - Unique Int64 SQL: **7.58% faster**. - Across 40 microbenchmark combinations: **1.79% faster** (equal-weight geometric mean). - All **20,500 allocation observations** retained zero estimation error, with unchanged allocation and capacity behavior. The major Int32 merge regression observed with earlier experimental approaches did not reproduce. The main SQL improvements were also confirmed in independent interleaved runs. **The inlining prototype is not part of this PR or a published hashbrown release.** I plan to validate its broader performance and compilation-cost impact before seeking upstream hashbrown review. The DataFusion PR will remain in Draft until the performance implications of the final implementation are resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
