andygrove commented on PR #2518: URL: https://github.com/apache/datafusion-ballista/pull/2518#issuecomment-5941481072
> > Requesting changes for two memory-scalability issues in the new coalescing path. The functional behavior and focused tests look good, but aggregate buffering is not bounded for high output-partition counts, and one input sequence can exceed the stated per-partition bound. > > Sorry for the trouble, I'll slow down and re-review. 👀 We went through similar experiments in Comet. We got better performance with coalescers per output partition but it didn't scale with high partition counts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
