ding-young commented on PR #17363: URL: https://github.com/apache/datafusion/pull/17363#issuecomment-3245248823
> BTW I can't predict how bad this round trip memory blowup issue gonna be, do you know any other? If it's common we might give up, and when this validation fail we can simply print some `warn!()` logs, and set some static max merge degree to finish the query. After this change, I believe all unit tests succeed with validation (within a 4KB diff). However, when the batch size is very large and the number of columns increases, the diff could potentially be larger. I’m not sure yet what a “safe” diff margin should be, and I haven’t found any other possible sources for the blowup so far. Given that, I think it’s better to print `warn!()` logs and recommend users to report their cases in the related issue, so we can investigate further if needed. I'll update the validation PR accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
