pepijnve commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2922419073
> However, the same example can also consume the pending return generated by Aggregate, so the guarantee cannot be upheld in our model. Indeed, which is why I've introduced cooperative yielding in the code in question in the meantime. I wonder if a more explicit graceful shutdown model for queries would be a more elegant solution to this problem. It kind of feels like every stream now needs to figure out how to make sure it yields itself rather than the framework solving this problem. Would some kind of cancellation token in TaskContext (something really simple like an AtomicBoolean wrapper) make more sense? That way streams would just need to call some `is_cancelled` method instead of counting batches, measuring time, etc. Top level you could still wrap everything in some drop wrapper that triggers the cancellation token. Has something like that ever been proposed (and maybe rejected)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
