ozankabak commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2929684773
@zhuqi-lucas, I think you are very close to a non-invasive solution that we can ship quickly 🚀 IIUC, this doesn't modify any operators, it just inserts a cancellation-friendly parent to all leaves. Three things come to mind as finishing touches: 1. Making the rule smarter, by utilizing `EmissionType` information, so that it only adds `YieldExec`s when necessary. If the path from the leaf to the root does not involve any operator that is pipeline-breaking, there is no need to insert a `YieldExec` as the parent of the leaf. I think this similar to @pepijnve' thinking. 2. Having a configuration flag to sidestep this rule for users who don't want to get any (however small) performance penalty. 3. Benchmarking to make sure performance penalty is small. Thanks for the awesome collaboration 💪 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
