joroKr21 commented on issue #9562: URL: https://github.com/apache/datafusion/issues/9562#issuecomment-2887207261
I reached the same conclusion. Had a hash join after aggregation and it wanted to allocated > 16GB memory for couple hundred MB of aggregation state 😢 > I think this approach sounds good -- nice proposal > One thing that could help keep the PRs small and manageable would be to switch the APIs as described above but you could avoid having to change all the GroupAccumulators in one PR by returning return a Vec<> of size 1 for most of them. > Then we can make subsequent PRs to switch over other groups accumulators as needed @alamb why are you talking about accumulator APIs? `ExecutionState::ProducingOutput` is purely local to `GroupedHashAggregateStream`. I already tried implementing this change and it worked great for my use case - the join no longer ran out of memory 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
