2010YOUY01 commented on PR #14411: URL: https://github.com/apache/datafusion/pull/14411#issuecomment-2646218873
> Hi @2010YOUY01. I'd like to thank you firstly for this investigation. I actually expect higher memory consumption—especially in systems where the upstream part of the RepartitionExec produces results faster than its downstream part (which is generally the case). But, as @Weijun-H asked, did you enable the configuration for on-demand repartitioning? I'm indirectly answering your second question below. Oops...no, I re-runned it and also saw similar memory usages, the result looks fine. The point is: I think there should be no memory-bloat issue in TPCH/clickbench queries caused by `RepartitionExec`, just wondering do you have any bad query can reproduce the memory issue reported by https://github.com/apache/datafusion/issues/14287, and will this PR fix it or it's only performance focused? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
