alamb commented on issue #24768: URL: https://github.com/apache/datafusion/issues/24768#issuecomment-5590357419
> A runtime fallback (hash join starts, memory runs out, switch to SMJ) is also possible, for every partition mode and every join type that SMJ can represent. > Two cases cannot use SMJ: null-aware anti join, and joins that promise probe-side ordering. Reasons below. I still think moving towards the runtime fallback would be valuable > 1. Null-aware anti join (x NOT IN (subquery)). SMJ cannot represent it. SortMergeJoinExec::try_new I think the right way to solve this is to teach SMJ how to handle null aware anti-join -- we could file a ticket and then anyone who actually had that usecase could potentially help solve it > 2. A join that promises probe-side ordering — Inner / Right / RightSemi / RightAnti / RightMark with an ordered probe child. > The guard is one line and changes no plan: refuse the fallback (fail exactly as today) when maintains_input_order()[1] is true and the probe child has an ordering. Every join without an ordering promise — the whole Left family, and the Right family over an unordered probe — can fall back freely. This makes sense. A small extension would be to extend the sort usage analysis (I think it is Pushdown sort https://github.com/apache/datafusion/blob/a5c809f98dedcf4b22f5d317ea8efe0720834925/datafusion/physical-optimizer/src/optimizer.rs#L176 from @zhuqi-lucas ) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
