d0ra-1h3-3xpl0ra opened a new pull request, #26830: URL: https://github.com/apache/datafusion/pull/26830
## Which issue does this PR close? Closes #26003. ## Rationale for this change `EXISTS`, `IN` and `NOT EXISTS` queries planned as sort-merge joins fail with "Unsupported data type in sort merge join comparator" on timezone-aware timestamp, `Dictionary`, `Decimal256` and `Time32`/`Time64` keys, once a key group runs to the end of a batch. Inner sort-merge join and hash join already work on these keys. ## What changes are included in this PR? The bitwise stream compares keys with `JoinKeyComparator` everywhere except in `keys_match`, the check for whether a key group continues into the next batch. That function still called `compare_join_arrays` for non-float keys, and `compare_join_arrays` only knows a fixed list of types. #25134 moved float keys over to `JoinKeyComparator`; this moves the rest. `keys_match` now builds a one-row comparator each time a key group reaches the end of a batch. The scan already rebuilds its comparators on every new batch, so I don't expect a measurable cost. One edge case changes: `compare_join_arrays` treated `DataType::Null` keys as always equal, while `JoinKeyComparator` treats them as unequal under `NullEqualsNothing`. As far as I can tell this can't change any output, because such keys never compare equal in the scan either, so the code paths that call `keys_match` don't run for them. `compare_join_arrays` has no callers left, but it's public in `joins::utils`, so I left it alone. Happy to deprecate it here or in a follow-up if you prefer. ## What is the testing strategy for this PR? - `sort_merge_join.slt`: the reproducer from the issue, plus `batch_size = 2` queries for `Dictionary`, `Decimal256` and `Time64` keys. - `join_key_type_batch_boundaries` in `sort_merge_join/tests.rs`: the same matrix as `join_float_key_batch_boundaries` (join type, null equality, sort direction, 1- and 2-row batches) for the five key types above. I moved the body of the float test into a shared helper so the two don't duplicate it. Like #25134, it's a physical-plan test, so batches split at exact rows regardless of what the optimizer does. Without the fix, all of the new cases fail with the error from the issue. The change also covers other types `compare_join_arrays` didn't handle (e.g. `Duration`, `Interval`, `Decimal32`/`Decimal64`, nested types), but I haven't added tests for those. ## Are there any user-facing changes? Those queries now return results instead of an error. No API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
