ariel-miculas commented on issue #24704: URL: https://github.com/apache/datafusion/issues/24704#issuecomment-5477588507
> After many experiment, I found blocked approach is only a better memory management approach to reduce the peak memory usage. It's also relevant for excessive spilling in the downstream operator, as described here: https://github.com/apache/datafusion/issues/22526#issuecomment-4568611822 Even more relevant for the [partitioning in datafusion-comet](https://github.com/apache/datafusion-comet/blob/main/native/shuffle/src/partitioners/multi_partition.rs#L102) because threre's a shuffle after the partial aggregation that will always hit this spilling issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
