pmoust commented on issue #26165: URL: https://github.com/apache/datafusion/issues/26165#issuecomment-6091903157
Checked on main (55.1.0). The new implementation keeps the emitted batch reserved. Only the legacy stream (`enable_migration_aggregate = false`) shrinks early. Unit test: `SUM(v) GROUP BY g`, 16,384 `Utf8View` keys, `batch_size = 1024`, `pool.reserved()` vs the first slice's `get_array_memory_size()`: | Path | Emitted | Reserved while slicing | After | |---|---|---|---| | default | 753,872 | 753,872 | 0 | | legacy | 753,872 | 165,128 | 248 | The legacy stream is being removed, so closing as fixed in 55. End behavior on 54.1: Q33 alone over 20M rows, allocated exceeded total pool reservation by ~1.2 GB for 60-80 ms at emit. The concurrent overshoot in the report also included allocator retention, so I don't have a clean "pool X, RSS Y" number for this bug alone. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
