Dandandan commented on issue #12454: URL: https://github.com/apache/datafusion/issues/12454#issuecomment-2351046713
> If there 1 huge table and another one is small, in this case the `broadcastJoin/mapJoin` works best. in distributed env(which is not DF) the big table repartitioned across nodes(every node can be DF), then small table copied to every node and usual hash join happens. The final result needs to be merged from nodes to the stream. This unfortunately is a bug that is present in Ballista as well, for left and full outer joins (every join type that produces rows for unmatched left side rows), a plain broadcast join won't work when planned on multiple nodes, I will file an issue on Ballista repo as well. Sharing the bitmap (to one node) has the least overhead, every other solution will require moving the full output to one node, so depends more on the join output size. FYI @andygrove -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
