zhuqi-lucas commented on PR #16398: URL: https://github.com/apache/datafusion/pull/16398#issuecomment-2988090319
> > Could share budget count affect performance? > > @zhuqi-lucas the situations where this can have an impact is when you have an operator that switches to a different input stream when the polled input stream returns pending. Interleave is a simple example of that. With the per stream budget, the second poll may end up returning ready. With the shared task budget the second poll will get 'budget depleted' and return pending as well. We may incur some overhead by attempting to poll the second stream even though you can already know that it will return pending. > > Besides that type of situation, and disregarding any non-obvious side effects of using thread locals for a moment, I don't see how shared vs non-shared budget would make a difference. There shouldn't be competition between tasks for the same budget; the counter is distinct per task and is reset each task 'tick'. Got it, so we add Cooperative to some operators such as sort_perserve_merging when partition > 1, it looks like similar to interleave which will have many inputs? I am not sure if it affects some performance. And the original solution, we only add yield to leaf node, leaf node will only has one input i believe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
