2010YOUY01 commented on PR #24785: URL: https://github.com/apache/datafusion/pull/24785#issuecomment-5495409196
Thanks for the detailed explanation. You're right that the comments are out of date; I'll fix them shortly. > So when the stream transitions back to `ReadingInput`, it is holding the full pre-OOM reservation for a table that is empty, and holds it until another input batch arrives — indefinitely if the input returns `Pending`. Instrumenting the last-slice branch with a growing-group-key fixture and a 6000-byte pool: If I understand correctly, the proposed alternative is: say the early emit happens in iteration K, then - current impl: memory is released in iteration K+1 - proposed alternative: memory is released in iteration K I don't think that amounts to a "definitely" vs. "indefinitely" difference, and the reproducer only checks whether memory can be released one iteration earlier, which isn't an end goal in itself. The bigger issue, I believe, is that we have to keep the large output batch in memory for a long time until it's finally dropped. I'd suggest revisiting the implementation once blocked memory management ships, to make it more memory-efficient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
