alamb commented on issue #16353: URL: https://github.com/apache/datafusion/issues/16353#issuecomment-2963580078
> Sorry, I usually do this around a white board. Yeah I agree doing this kind of thing with a whiteboard is easier. I am happy to set up a video call with the relevant parties if that might be more efficient (and it would be nice to meet you all "face to face") > First and foremost I'm trying to communicate what I see as intrinsic problems with the external optimizer rule approach. This was countered with a belief that it can be solved, but I don't see it yet myself. Indeed -- it is my perspective that 1. https://github.com/apache/datafusion/pull/16196 handles most, but not all, cases where an `Stream` implementation may not yield directly. 2. For some of the more advanced cases you describe (like switching a stream mid-execution) the general optimizer approach will not permit cancelling and thus the individual stream implementations need to be made yield aware. > stream in a YieldStream". [#16301](https://github.com/apache/datafusion/pull/16301) does not make use of Tokio's coop budget yet however. Instead it's the per-operator counter variant. The diff per operator does illustrate the scope and nature of the change `Stream` implementers would need to make. This makes sense (and I am sure we could encapsulate the changes from 16301 more too if we go with that approach) I wonder if your main concern is that the optimizer approach we merged does not handle all of the cases, so it potentially leaves a trap for future Stream implementors (who do more avanced streams)? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
