zhuqi-lucas commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2942575262
> > What do you mean built-in yielding? If it means we add some leaf nodes to support built-in yielding support? Thanks! > > Here is rough sketch of what I meant, let's discuss and optimize. > > 1. We add built-in yielding capabilities to `DataSourceExec` via a constructor argument; i.e. it yields internally every now and then if created with that constructor argument. > 2. We add two `ExecutionPlan` APIs: > > * `yields_cooperatively` (we can choose the name later, this is a placeholder), which returns a boolean value. The default implementation simply checks `EmissionType` of the operator and returns accordingly. > * `with_cooperative_yields`, which _optionally_ returns a cooperatively-yielding version of the operator if it exists. The default implementation checks `yields_cooperatively` and returns self if true, and `None` otherwise. > > 3. We override `with_cooperative_yields` for `DataSourceExec` and return the yielding version. > 4. If the rule decides that it needs a leaf to yield, it checks whether a yielding variant exists, and replaces it with a yielding variant if it does. If it doesn't, it adds a `YieldStreamExec`. > > After our investigations, if it turns out that `InterleaveExec` or some other operator can always misbehave, we can change the return value of `yields_cooperatively` to signal that and tweak the rule to add intermediate `YieldStreamExec` nodes too. > > @zhuqi-lucas, I may be missing some detail, but does it make sense from a general point of view? Good example, thank you @ozankabak ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
