coderfender opened a new issue, #6514: URL: https://github.com/apache/datafusion-comet/issues/6514
### Describe the enhancement requested PR #4587 adds native execution for **hash** existence joins (BroadcastHashJoin / ShuffledHashJoin), but existence **sort-merge joins** still fall back to Spark. Spark plans a sort-merge join for an `EXISTS` / `IN`-combined-with-`OR` subquery whenever the subquery side is too large to broadcast, so this is the biggest remaining gap for native existence-join coverage after #4587. ### Blocker DataFusion 55.1's `BitwiseSortMergeJoinStream` (the `LeftMark` SMJ path) buffers completed output inside its spanning-key / exhausted-inner loops and only flushes via `emit_completed_batches` after those loops return. The memory reservation tracks the inner key buffer, not that queued output, so a large equal-key group or an empty right input can retain an entire left partition outside the configured memory budget before the first output batch is emitted — risking executor OOM even though the join reports little reserved memory. Found by @sunchao: https://github.com/apache/datafusion-comet/pull/4587#discussion_r4042586365 ### Suggested direction Enable `CometSortMergeJoinExec` for `ExistenceJoin` once the DataFusion mark-SMJ stream emits completed output incrementally inside those loops (and the reservation accounts for queued output). A regression should verify incremental output and bounded retained memory for both empty-right and skewed-equal-key inputs. ### Context Split out from PR #4587 (native ExistenceJoin). The fallback is intentional for now; the comment in `CometSortMergeJoinExec.convert` links here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
