pepijnve commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2939714394
> I think it looks good, but it also seems like it doesn't fundamentally change @ozankabak 's concern of having to annotate all streams with cancel aware behavior 🤔 That's correct indeed. What I'm going for is to make the necessary code changes as simple as possible and keep overhead minimal. I agree with you about needing finer control than the current PR provides wrt the yield penalty with the current approach. For a hash join for instance (and most joins), you want the yield logic on the build side for instance. But on the probe side you most likely do not. `EmissionType` does not provide sufficient detail to describe this. I'm not convinced yet that adding yield points where appropriate is going to be that big of an issue which is why I started working on this branch. I wanted to get a first version out to solicit feedback as soon as possible. I'm still working on going through the other operators. Basically what you need to look out for, and what could be fairly easy to write some guidelines on, is the presence of looping constructs in a stream's `poll_next`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
