phillipleblanc commented on PR #2188:
URL:
https://github.com/apache/datafusion-ballista/pull/2188#issuecomment-5092436844
> Thanks for picking this up. The hang is real, and the
`hash_join_swap_subrule` change matches upstream DataFusion exactly, which is
nice.
Thanks, @andygrove. You were right. I reproduced the 56/75-row incorrect
results with your test and revised the PR.
The updated implementation now:
- keeps null-aware joins as unswapped LeftAnti(CollectLeft);
- coalesces the probe side so exactly one task executes the join;
- applies this lowering in both static planning and AQE;
- performs AQE coalescing in DistributedExchangeRule, after DataFusion's
optimizer rules;
- restores the vendored Partitioned branch to match upstream DataFusion;
- rejects a disabled broadcast threshold or a build side known to exceed
it. Unknown sizes remain allowed because file scans commonly lose exact
statistics before lowering.
I adapted your standalone test to cover `prefer_hash_join` and AQE with four
input partitions, both with and without a subquery NULL. All cases now match
single-process DataFusion: 0 and 15 rows respectively.
The default sort-merge path still loses the null-aware flag before scheduler
planning. I filed #2193 for that separate issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
