adriangb opened a new issue, #25779:
URL: https://github.com/apache/datafusion/issues/25779
### Describe the bug
A Parquet scan with `datafusion.execution.parquet.pushdown_filters = false`
(the default) can return rows in the wrong order, and `ORDER BY ... LIMIT` can
return wrong rows.
With `pushdown_filters = false`, `ParquetSource` keeps a pushed-down
predicate only for row group and page pruning, and it replies `PushedDown::No`,
so a `FilterExec` stays above the scan. But `FileScanConfig::eq_properties`
reads the same predicate through `FileSource::filter()` and adds its equalities
to the scan's equivalence properties. The scan then claims that `a` is constant
(for `a = 5`) or that `a` and `b` are equal (for `a = b`), while it emits rows
that violate this.
An operator placed directly on the scan, below the `FilterExec`, trusts the
false property. An order-preserving `RepartitionExec` takes its merge keys from
the scan's normalized output ordering. Normalization removes the "constant"
`a`, so the repartition merges streams that are sorted by `(a, b)` using `b`
only. The `FilterExec` then removes the `a != 5` rows, and the rows that stay
are out of order.
### To Reproduce
```sql
COPY (SELECT * FROM (VALUES (3, 9), (5, 1)) v(a, b)) TO 'mre/f1.parquet'
STORED AS PARQUET;
COPY (SELECT * FROM (VALUES (5, 2)) v(a, b)) TO 'mre/f2.parquet' STORED AS
PARQUET;
CREATE EXTERNAL TABLE t (a INT, b INT) STORED AS PARQUET LOCATION 'mre/'
WITH ORDER (a ASC, b ASC);
SET datafusion.execution.target_partitions = 4;
SET datafusion.optimizer.repartition_file_scans = false;
-- Expected (5,1,1), (5,2,1). Actual: (5,2,1), (5,1,1)
SELECT a, b, count(*) FROM t WHERE a = 5 GROUP BY a, b ORDER BY b;
SET datafusion.optimizer.prefer_existing_sort = true;
-- Expected 1. Actual: 2
SELECT b FROM t WHERE a = 5 ORDER BY b LIMIT 1;
```
The plan of the `LIMIT` query shows the mismatch. The scan declares
`output_ordering=[a, b]`, but the repartition merges on `b` only:
```
SortPreservingMergeExec: [b@0 ASC NULLS LAST], fetch=1
FilterExec: a@0 = 5, projection=[b@1], fetch=1
RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2,
preserve_order=true, sort_exprs=b@1 ASC NULLS LAST
DataSourceExec: file_groups={2 groups: [...]}, projection=[a, b],
output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], file_type=parquet,
predicate=a@0 = 5, ...
```
The equality form fails in the same way. With files sorted by `a`, `WHERE a
= b ORDER BY a` returns `5, 3` instead of `3, 5`, because the repartition
merges on `b`.
### Expected behavior
The results are the same as with `pushdown_filters = true`, which is correct
in all of these cases. DuckDB gives the same results.
### Additional context
- Reproduced on `main` at e8e41ae958.
- With the default configuration, the scan is split into `target_partitions`
file groups, so no `RoundRobin` repartition is added below the `FilterExec`.
`repartition_file_scans = false` is sufficient to trigger the bug. Declared
file partitioning (`preserve_file_partitions`) should also trigger it, but I
did not test that.
- The equivalence derivation was added in
https://github.com/apache/datafusion/pull/16686 (for
https://github.com/apache/datafusion/issues/16563). It is correct when
`pushdown_filters = true`, because then the scan applies the filter to every
row and the `FilterExec` is removed.
- Operators that filter pushdown moves through (aggregate, sort, projection,
join, union) are not affected. They commute with the filter, and a property
that holds on the filtered rows is sufficient for them. The failure needs a
node that is inserted directly on the scan after pushdown.
- Dynamic filters (hash join, TopK) are not affected. They are not plain `=`
expressions, and the equivalences are computed at planning time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
