ctsk opened a new pull request, #15479: URL: https://github.com/apache/datafusion/pull/15479
Relates to Issue: #15478 ## Rationale for this change The blocking operators (HJ buid side, Aggregation) are often planned on top of a RepartitionExec with a CoalesceBatchesExec in-between. However, one of the first things these operators do is concatenate the freshly CoalescedBatches. This PR is to test if the overhead of the 2-step coalesce+concat outweighs the gains of fewer dispatches of the consuming operators. ## What changes are included in this PR? This PR adds a physical optimizer rule `UncoalesceBatches`. It runs after the `CoalesceBatches` rule and removes `CoalesceBatchesExec` that are at the build side of HashJoins and in front of non-partial aggregations ## Are these changes tested? Not yet! ## Are there any user-facing changes? Yes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
