pepijnve commented on PR #16196: URL: https://github.com/apache/datafusion/pull/16196#issuecomment-2949117788
> Tests pass even if add in the "pretending" (because the join code seems to yield naturally) The hash join test I have does fail so I dug into this. It's passing for you for two reasons: - There's an aggregation in that test, so the plan has emission type final and the yield wrapper gets injected - If you remove the aggregation there's still a RepartitionExec which itself has an ad hoc version of the cooperative yield logic in place at https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/repartition/mod.rs#L980. It also decouples producer and consumer via the distribution channels. If the consumer drains faster than the producer fills you'll get a natural pending that way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
