dwsmith1983 commented on PR #5565: URL: https://github.com/apache/datafusion-comet/pull/5565#issuecomment-5582668040
The multi-partition path builds its writers one at a time, so "ten thousand retained builders" was wrong; the effect is per flush, not simultaneous retention. Measured with the allocation counter #5727 added, driving `LocalPartitionWriter` in multi mode over 10,000 partitions with a 300-row batch, finish pass and spill pass, warm-up first, main at bb9e74020 versus this head: | codec | main allocs / bytes per flushed partition | head | delta | |---|---|---|---| | none | 48.0 / 36,648 | 40.0 / 36,096 | -8 / -552 | | lz4 | 51.0 / 190,677 | 43.0 / 190,125 | -8 / -552 | | zstd 3 | 49.0 / 69,416 | 41.0 / 68,864 | -8 / -552 | The 8 allocations and 552 bytes are the `FlatBufferBuilder` growing during its first message, once per fresh context, and the same across codecs, so that part is the Arrow context and nothing else; it grows with message width (12 / 8,232 at 50 columns). Empty partitions cost nothing on either tree. The counter cannot see the zstd C side: main creates a `CCtx` per flushed partition, about 3.7 MB at level 3 by `sizeof`, where the head creates one per task, which is the larger effect and also a count effect. Probe sources and raw output are in the discussion if wanted; the description now carries the Linux run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
