Kristofferjoha commented on issue #27323: URL: https://github.com/apache/datafusion/issues/27323#issuecomment-6819294776
@adriangb Dug into where the ~174 KB batches come from. `ByteViewGroupValueBuilder::take_n` copies the in-progress buffer from the start on every partial emit and never trims it, so each batch drags along the strings of all the batches before it. #27360 fixes that. With it the repro passes at "-m 10M", and the partial aggregate output drops from 42.6 MB to roughly 1 MB (repartition goes from 330 MB to 5.5 MB). That only fixes the trigger though. The merge still holds every input batch until it has a full output batch, and the repartition still charges each partition for the whole source batch's buffers. For the merge part, do you want to go with the early emit in BatchBuilder, or try setting `with_max_batch_bytes` on the repartition merge first? Looks like the sort spill merge already uses it, but I haven't tried it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
