txwyy123 commented on PR #26149: URL: https://github.com/apache/datafusion/pull/26149#issuecomment-6174410204
Good call on `column_count/64`. I investigated this: **Root cause**: The overflow `HashMap<NonZero<usize>, u32>` stores 12 bytes per entry (8-byte key + 4-byte count) vs the original `HashSet<NonZero<usize>>` at 8 bytes per entry. With 64 unique buffers all in the HashMap, that is 50% more memory per bucket → worse cache line utilization on every lookup. The `entry().or_insert(0)` API path is already optimal for hashbrown; the cost is structural — refcounting requires storing a count alongside each key. **Options I see:** 1. **Accept +7% (+144ns) on the overflow path** — `column_count/64` exercises 64 *unique* buffers (one per column, no sharing). In practice operators work with sliced batches (shared buffers), which is the `shared_slices` benchmark — and that path is 2–14% *faster* than main. The >16-unique-buffer case is uncommon in real operator workloads. 2. **Reduce to 8 bytes per entry** — store only a `u64` in the overflow set, packing the count into the upper bits (`(id << 32) | count`). This restores the original cache footprint but makes the code less readable and limits the count to 32 bits (already the case). 3. **Use two parallel `HashSet` + `HashMap`** — `HashSet` for the fast "is-this-new?" check, `HashMap` only for entries with count > 1. Adds code complexity but keeps the common (count=1) path at `HashSet` speed. I lean toward option 1 — the +7% is on the cold/rare path and the hot path (`shared_slices`) improved. But happy to implement option 2 or 3 if you prefer. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
