cool9850311 opened a new issue, #27494:
URL: https://github.com/apache/datafusion/issues/27494
### Describe the bug
`file_row_index()` and `input_file_name()` fail at execution when a query has
both a skip and a fetch (`OFFSET n LIMIT m`) against a source that supports
skip
pushdown. Either clause on its own is fine.
A source that supports skip pushdown wraps its scan in a `GlobalLimitExec`
for
the skip (`datafusion/catalog-listing/src/table.rs:814`). That node lands
between the scan and the projection the leaf-extraction rules placed
directly on
it, and nothing moves the projection back down past it, so the marker is
evaluated above the scan — where it cannot be resolved.
### To Reproduce
`datafusion-cli`, stock configuration, run in a directory containing an empty
`offset_repro/`.
```sql
COPY (SELECT value AS column1 FROM range(0, 5))
TO 'offset_repro/small.parquet' STORED AS PARQUET;
CREATE EXTERNAL TABLE small
STORED AS PARQUET LOCATION 'offset_repro/small.parquet';
SELECT file_row_index() FROM small LIMIT 2; -- ok: 0, 1
SELECT file_row_index() FROM small OFFSET 1; -- ok: 1, 2, 3, 4
SELECT file_row_index() FROM small OFFSET 0 LIMIT 2; -- ok: 0, 1
SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
-- Execution error: file_row_index() is source dependent and cannot be
evaluated directly
SELECT length(input_file_name()) > 0 FROM small OFFSET 1 LIMIT 2;
-- Execution error: input_file_name() is source dependent and cannot be
evaluated directly
```
Five rows, one file, one expression in the projection.
### Expected behavior
| Query | DataFusion returns | Expected |
|---|---|---|
| `SELECT file_row_index() FROM small OFFSET 1 LIMIT 2` | `Execution error:
file_row_index() is source dependent and cannot be evaluated directly` | `1`,
`2` |
| `SELECT length(input_file_name()) > 0 FROM small OFFSET 1 LIMIT 2` | the
same error, for `input_file_name()` | `true`, `true` |
### Additional context
<details>
<summary>EXPLAIN: the marker is left above the limit, and the scan has no
rewrite</summary>
```text
-- SELECT file_row_index() AS r FROM small OFFSET 1 LIMIT 2
ProjectionExec: expr=[file_row_index() as r]
GlobalLimitExec: skip=1, fetch=2
DataSourceExec: file_groups={1 group: [[small.parquet]]}, limit=3,
file_type=parquet
-- SELECT file_row_index() AS r FROM small LIMIT 2
DataSourceExec: file_groups={1 group: [[small.parquet]]},
projection=[CAST(__datafusion_file_row_index@1 AS Int64;N) as r], limit=2,
file_type=parquet
```
Without the skip the source absorbs the whole projection and rewrites the
marker
into a column. With it, the projection is stranded above the limit and the
marker
is evaluated there.
</details>
- The skip reaches the scan through `skip > 0 &&
scan.source.supports_skip_pushdown()`
(`datafusion/optimizer/src/push_down_limit.rs:112`), and `GlobalLimitExec` has
no `try_swapping_with_projection`, so `ProjectionPushdown` cannot move the
projection back below it.
- With more than one file group, `EnsureRequirements` adds a
`CoalescePartitionsExec` under that limit, so two nodes sit between the scan
and the projection rather than one.
- Neither `batch_size` (8192 and 30000), `target_partitions` (8 and 1),
table size (5 rows and 20,000) nor `parquet.pushdown_filters` (on and off)
changes the outcome, and there is no `RepartitionExec` in these plans. That
distinguishes this from the shape fixed by #27152, which needed both a
`RepartitionExec` and a second, non-leaf expression in the same projection.
- Fix direction: split a projection that mixes a leaf expression with one
that must stay in place, and send only the leaf half below the limit. PR #27395
does this. On that branch a bare marker with `OFFSET n LIMIT m` returns rows
instead of the error, for both `file_row_index()` and `input_file_name()`.
- Split out of #27131 at the reviewer's request; the measurements there came
from the same runs.
- Tested on `main` at `1841fb6ec`; line numbers are from that commit.
### Willingness to contribute
I plan to contribute a fix for this bug independently (PR #27395).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]