buraksenn opened a new pull request, #24298: URL: https://github.com/apache/datafusion/pull/24298
## Which issue does this PR close? - Closes #24297 ## Rationale for this change `ClassicPWMJStream` transitions phases while its output `BatchCoalescer` still holds completed batches. The Left/Full unmatched pass falls back into its producer when the drained queue is an exact multiple of `batch_size` — emitting duplicate rows forever — and queued unmatched Right/Full batches are dropped at terminal transitions. A zero-row placeholder batch also escapes as stream output on empty results. ## What changes are included in this PR? - Add `BatchProcessState::next_drained_batch()` (`finish_buffered_batch` + `next_completed_batch`); `None` guarantees the coalescer is empty - `process_stream_batch` and `process_unmatched_buffered_batch` drain one batch per poll and transition only after a confirmed-empty pop, so a finished producer can never re-run and no queued batch is lost - Scan completion returns `StatefulStreamResult::Continue` instead of the empty placeholder batch; empty joins now yield no batches ## Are these changes tested? Yes. The existing unmatched Left/Right tests now run with `batch_size = 1` (the Left one bounded with `take(6)` so the old duplicate loop fails as a snapshot mismatch instead of hanging) — both fail on main and pass here. The empty-join tests assert no batches are emitted. ## Are there any user-facing changes? No (PWMJ is experimental and off by default). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
