cetra3 opened a new issue, #27318:
URL: https://github.com/apache/datafusion/issues/27318
### Describe the bug
This query fails to plan on `main`:
```sql
SET datafusion.execution.target_partitions = 2;
SET datafusion.optimizer.hash_join_single_partition_threshold_rows = 0;
COPY (SELECT named_struct('k', 'a') AS s) TO 'data/t/1.parquet';
COPY (SELECT named_struct('k', 'b') AS s) TO 'data/t/2.parquet';
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'data/t/';
SELECT s['k'] FROM t WHERE s['k'] = 'a' OR s['k'] IN (SELECT s['k'] FROM t)
GROUP BY 1;
```
```text
SanityCheckPlan: order and distribution requirements stopped holding after
physical optimizer rule 'ProjectionPushdown'
caused by
Error during planning: Plan: [
AggregateExec: mode=SinglePartitioned, gby=[__datafusion_extracted_2@0 as
__datafusion_extracted_2], aggr=[]
FilterExec: __datafusion_extracted_3@1 = a OR mark@0,
projection=[__datafusion_extracted_2@2]
ProjectionExec: expr=[mark@0 as mark, get_field(s@1, k) as
__datafusion_extracted_3, get_field(s@1, k) as __datafusion_extracted_2]
HashJoinExec: mode=Partitioned, join_type=LeftMark,
on=[(__datafusion_extracted_4@0, t.s[k]@0)], projection=[mark@2, s@1]
RepartitionExec: partitioning=Hash([__datafusion_extracted_4@0], 2),
input_partitions=2
DataSourceExec: file_groups={...}, projection=[get_field(s@0, k)
as __datafusion_extracted_4, s], file_type=parquet
RepartitionExec: partitioning=Hash([t.s[k]@0], 2), input_partitions=2
DataSourceExec: file_groups={...}, projection=[get_field(s@0, k)
as t.s[k]], file_type=parquet
] does not satisfy distribution requirements:
KeyPartitioned[[__datafusion_extracted_2@0]]). Child-0 output partitioning:
Hash([__datafusion_extracted_4@0], 2)
```
The settings make the join `Partitioned` on this small data set.
### Expected behavior
The query returns `a` and `b`, as it does at `0d3aac0bde`.
### Additional context
A bisect points to #26747 (`99e723dd47`). Its parent `ce0e8293f9` plans the
query.
#26747 records `get_field(s, k) ≡ __datafusion_extracted_4` when projecting
equivalences. That lets `EnsureRequirements` use the join's hash partitioning
for the group key and skip a repartition, which is correct. Then
`ProjectionPushdown` moves `get_field(s, k)` into the `DataSourceExec`
projection and drops `__datafusion_extracted_4` through the join's embedded
projection. Neither step records the equivalence again. So the join's output
partitioning becomes an unknown column, and the final check rejects the plan.
No rows move in that rewrite, so the query fails to plan and never returns
wrong results.
---
Claude Code (an AI coding assistant) investigated this and wrote this issue
on behalf of cetra3. A PR with a fix follows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
