bvolpato opened a new pull request, #23469:
URL: https://github.com/apache/datafusion/pull/23469
## Which issue does this PR close?
- Closes #18066.
## Rationale for this change
When physical uncorrelated scalar subqueries are disabled, DataFusion
rewrites a scalar subquery in a projection into a left join. Optimization can
remove its always-true join filter, so the Substrait producer emits a `JoinRel`
without an expression.
The consumer passed that missing expression through as no filter. DataFusion
rejects non-inner joins without a condition, returning `Plan("join condition
should not be empty")` instead of consuming the plan.
This supersedes stale #18068. Unlike that earlier patch, this change does
not convert conditionless inner joins to cross joins, addressing its review
feedback.
## What changes are included in this PR?
- Use `ON true` when consuming a non-inner `JoinRel` with no expression.
- Preserve existing conditionless inner-join behavior.
- Add an end-to-end scalar-subquery projection regression that forces the
join rewrite, verifies the consumed plan, and executes the result.
## Are these changes tested?
Yes.
- Regression test fails before the fix with `Plan("join condition should not
be empty")` and passes after it.
- `cargo test -p datafusion-substrait --features protoc`: 49 unit, 201
integration, and 3 doc tests passed; 6 existing tests ignored.
- `cargo clippy --all-targets --all-features -- -D warnings`: passed.
- `PROTOC=<vendored protoc> ./dev/rust_lint.sh`: passed, including fmt, CI
clippy, TOML, license, typos, Prettier, and rustdoc.
- Extended workspace test: 453/453 DataFusion library tests passed and
1027/1028 core integration tests passed. Unrelated `sort_with_mem_limit_1`
failed its host RSS ceiling (217.3 MB measured, 190.7 MB limit), matching the
existing local baseline.
## Are there any user-facing changes?
Yes. Substrait roundtrips no longer reject scalar subqueries in projections
when they are rewritten as conditionless left joins. No public API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
