andygrove opened a new issue, #1448: URL: https://github.com/apache/datafusion-comet/issues/1448
### Describe the bug As demonstrated in the new unit tests added in https://github.com/apache/datafusion-comet/pull/1440, native shuffle is double allocating memory. ```rust // TODO: note that we are currently double-counting the memory usage // because we reserve the memory twice - once at the repartitioner level // and then again in each PartitionBuffer assert_eq!(212992, repartitioner.reservation.size()); assert_eq!( 106496, repartitioner.buffered_partitions[0].reservation.size() ); assert_eq!( 106496, repartitioner.buffered_partitions[1].reservation.size() ); ``` ### Steps to reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
