zhuqi-lucas commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2924691085
> but it's kind of similar to the count-batches-and-yield solution. Thank you @pepijnve , this is really helpful, so our choice seems like good enough: 1. For performance, we no regression. 2. For usage, we fix the bug. > Just as an FYI, I was reading through the tokio docs yesterday and found https://docs.rs/tokio/latest/tokio/task/coop/index.html#cooperative-scheduling which seems very similar to the situation we're seeing here with pipeline-breaking operators. The solution provided by https://docs.rs/tokio/latest/tokio/task/coop/fn.consume_budget.html is not directly applicable for Stream implementations since it's an async function, but it's kind of similar to the count-batches-and-yield solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
