pepijnve commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2924937197
> This is an interesting find, and I think it would be great if we could somehow utilize this (or something similar to this). I was just studying the implementation. It would be interesting to use since tokio is setting up the budget counter already anyway, but it's all inaccessible at the moment except via that async function. I found some pointers online on how to call async code from a Future. It seems feasible, but not very ergonomic. In the end the tokio budget is not much more than a thread-local counter though. Calling `consume_budget` decrements the counter. If the counter reaches zero, the waker is registered, and `Pending` is returned. The next time the task is scheduled tokio gives it a fresh budget (hardcoded to 128 at the moment afaict). So this is essentially the same approach as was proposed with `YieldStream`. The main difference is that the yield/budget counter is global for the task instead of being tracked per operator at each level. Not sure if that's worth it the overhead of accessing thread-local storage or introducing an additional per-spawned-task context to hold this counter. One possible improvement over the `YieldStream` decorator (convenient as it may be) might be to split off the counter itself into something tokio `Budget` like. `YieldStream` would then just be a convenience wrapper. That way operators can be more explicit about when they 'consume budget' rather than hiding it in the poll of the input stream. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
