paul-heyse opened a new issue, #25978:
URL: https://github.com/apache/datafusion/issues/25978
### Describe the bug
A `GROUP BY` aggregate over a `UNION ALL` fails to plan when one branch
projects
`coalesce(<nullable boolean expression>, FALSE)`:
```
Internal error: Physical input schema should be the same as the one
converted from logical input schema. Differences:
- field nullability at index 1 [f]: (physical) true vs (logical) false.
This issue was likely caused by a bug in DataFusion's code. Please help us
to resolve this by filing a bug report in our issue tracker:
https://github.com/apache/datafusion/issues
```
The optimizer rewrites `coalesce(x = 'y', FALSE)` into `__common_expr_1 IS
NOT NULL AND __common_expr_1`
(plan below). The rewritten expression is physically nullable, while the
`Union`'s logical schema keeps
the non-nullable type of `coalesce(..., FALSE)`, so the aggregate's schema
check fails.
Reproduces on **datafusion-cli 55.1.0** (latest crates.io release) and
53.1.0, and on the
**datafusion 54.0.0** Python wheel.
### To Reproduce
```sql
CREATE TABLE a (k VARCHAR NOT NULL, x VARCHAR) AS VALUES ('k1', 'y'), ('k2',
NULL);
CREATE TABLE b (k VARCHAR NOT NULL) AS VALUES ('k3');
SELECT k, bool_or(f) AS f FROM (
SELECT k, coalesce(x = 'y', FALSE) AS f FROM a
UNION ALL
SELECT k, TRUE AS f FROM b
) u GROUP BY k;
```
`datafusion-cli -f repro.sql` raises the internal error above.
Variants, measured on 55.1.0:
| Variant | Result |
|---|---|
| As above | **Internal error** |
| The two branches swapped | **Internal error** |
| No `UNION` (the `coalesce` branch alone), same `GROUP BY k` | OK |
| No aggregate above the `UNION` | OK |
| Global aggregate (`count(f)` without `GROUP BY`) | OK |
| `coalesce(n, 0)` on an `INT` column, or `coalesce(x, 'z')` on the text
column | OK |
| `x IS NOT NULL` instead of the `coalesce` | OK |
The plan without the aggregate (`EXPLAIN SELECT k, f FROM (...) u`,
`datafusion.explain.format = 'indent'`):
```
logical_plan SubqueryAlias: u
Union
Projection: a.k, __common_expr_1 IS NOT NULL AND
__common_expr_1 AS f
Projection: a.x = Utf8View("y") AS __common_expr_1, a.k
TableScan: a projection=[k, x]
Projection: b.k, Boolean(true) AS f
TableScan: b projection=[k]
physical_plan UnionExec
ProjectionExec: expr=[k@1 as k, __common_expr_1@0 IS NOT
NULL AND __common_expr_1@0 as f]
ProjectionExec: expr=[x@1 = y as __common_expr_1, k@0 as k]
DataSourceExec: partitions=1, partition_sizes=[1]
ProjectionExec: expr=[k@0 as k, CAST(true AS Boolean) as f]
DataSourceExec: partitions=1, partition_sizes=[1]
```
With `SET datafusion.optimizer.max_passes = 0` the same aggregate query
fails differently:
`Internal error: coalesce should have been simplified to case.`
### Expected behavior
The query plans and returns `k1 → true`, `k2 → false`, `k3 → true`.
### Additional context
- Possibly the same family as #25369 (a CSE-extracted expression with
mismatched nullability
under an aggregate), but the trigger differs: the SQL contains no
duplicated expression, the
failure needs a `UNION` and a `GROUP BY`, and the mismatched field is the
projected column `f`
rather than `__common_expr_1`. The fix in progress there (#25684)
addresses `CASE` conditions over
several columns.
- Found while generating aggregate queries over unions of per-table rows. We
work around it by
materializing the union before aggregating.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
