haohuaijin opened a new issue, #25263:
URL: https://github.com/apache/datafusion/issues/25263
### Describe the bug
Enabling join dynamic filter pushdown can silently discard a matching row
when the filter passes through a `ProjectionExec` whose output has two columns
with the same alias (here `a.id` and `b.id`, both named `id`).
With the same data and SQL, disabling
`datafusion.optimizer.enable_join_dynamic_filter_pushdown` returns one row;
enabling it returns zero rows, without an error.
### To Reproduce
Run the following SQL in a fresh `datafusion-cli` session. The two `/tmp/`
Parquet paths must not already exist.
```sql
SET datafusion.optimizer.join_reordering = false;
COPY (SELECT 'a1' AS id, 'x1' AS ty)
TO '/tmp/df_dynamic_filter_projection_a.parquet'
STORED AS PARQUET;
COPY (SELECT 'x1' AS id)
TO '/tmp/df_dynamic_filter_projection_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_projection_b.parquet';
-- Returns one row: (a1, a1, x1, x1!).
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
SELECT a.id, b.id, a.ty || '!' AS tag
FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;
-- Incorrectly returns zero rows.
SET datafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECT s.id, a.id, b.id, tag
FROM ta s
JOIN (
SELECT a.id, b.id, a.ty || '!' AS tag
FROM ta a JOIN tb b ON a.ty = b.id
)
ON s.id = a.id;
```
### Expected behavior
_No response_
### Additional context
`ProjectionExec` resolves pushed-down filter columns by output alias instead
of by position, so with two outputs named `id` the filter on `b.id` is
rewritten to `a.id`. Found while working on #25244; the fix is included in
#25259.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
