cool9850311 opened a new issue, #27494:
URL: https://github.com/apache/datafusion/issues/27494

   ### Describe the bug
   
   `file_row_index()` and `input_file_name()` fail at execution when a query has
   both a skip and a fetch (`OFFSET n LIMIT m`) against a source that supports 
skip
   pushdown. Either clause on its own is fine.
   
   A source that supports skip pushdown wraps its scan in a `GlobalLimitExec` 
for
   the skip (`datafusion/catalog-listing/src/table.rs:814`). That node lands
   between the scan and the projection the leaf-extraction rules placed 
directly on
   it, and nothing moves the projection back down past it, so the marker is
   evaluated above the scan — where it cannot be resolved.
   
   ### To Reproduce
   
   `datafusion-cli`, stock configuration, run in a directory containing an empty
   `offset_repro/`.
   
   ```sql
   COPY (SELECT value AS column1 FROM range(0, 5))
   TO 'offset_repro/small.parquet' STORED AS PARQUET;
   
   CREATE EXTERNAL TABLE small
   STORED AS PARQUET LOCATION 'offset_repro/small.parquet';
   
   SELECT file_row_index() FROM small LIMIT 2;             -- ok: 0, 1
   SELECT file_row_index() FROM small OFFSET 1;            -- ok: 1, 2, 3, 4
   SELECT file_row_index() FROM small OFFSET 0 LIMIT 2;    -- ok: 0, 1
   
   SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
   -- Execution error: file_row_index() is source dependent and cannot be 
evaluated directly
   
   SELECT length(input_file_name()) > 0 FROM small OFFSET 1 LIMIT 2;
   -- Execution error: input_file_name() is source dependent and cannot be 
evaluated directly
   ```
   
   Five rows, one file, one expression in the projection.
   
   ### Expected behavior
   
   | Query | DataFusion returns | Expected |
   |---|---|---|
   | `SELECT file_row_index() FROM small OFFSET 1 LIMIT 2` | `Execution error: 
file_row_index() is source dependent and cannot be evaluated directly` | `1`, 
`2` |
   | `SELECT length(input_file_name()) > 0 FROM small OFFSET 1 LIMIT 2` | the 
same error, for `input_file_name()` | `true`, `true` |
   
   ### Additional context
   
   <details>
   <summary>EXPLAIN: the marker is left above the limit, and the scan has no 
rewrite</summary>
   
   ```text
   -- SELECT file_row_index() AS r FROM small OFFSET 1 LIMIT 2
   ProjectionExec: expr=[file_row_index() as r]
     GlobalLimitExec: skip=1, fetch=2
       DataSourceExec: file_groups={1 group: [[small.parquet]]}, limit=3, 
file_type=parquet
   
   -- SELECT file_row_index() AS r FROM small LIMIT 2
   DataSourceExec: file_groups={1 group: [[small.parquet]]}, 
projection=[CAST(__datafusion_file_row_index@1 AS Int64;N) as r], limit=2, 
file_type=parquet
   ```
   
   Without the skip the source absorbs the whole projection and rewrites the 
marker
   into a column. With it, the projection is stranded above the limit and the 
marker
   is evaluated there.
   
   </details>
   
   - The skip reaches the scan through `skip > 0 && 
scan.source.supports_skip_pushdown()` 
(`datafusion/optimizer/src/push_down_limit.rs:112`), and `GlobalLimitExec` has 
no `try_swapping_with_projection`, so `ProjectionPushdown` cannot move the 
projection back below it.
   - With more than one file group, `EnsureRequirements` adds a 
`CoalescePartitionsExec` under that limit, so two nodes sit between the scan 
and the projection rather than one.
   - Neither `batch_size` (8192 and 30000), `target_partitions` (8 and 1), 
table size (5 rows and 20,000) nor `parquet.pushdown_filters` (on and off) 
changes the outcome, and there is no `RepartitionExec` in these plans. That 
distinguishes this from the shape fixed by #27152, which needed both a 
`RepartitionExec` and a second, non-leaf expression in the same projection.
   - Fix direction: split a projection that mixes a leaf expression with one 
that must stay in place, and send only the leaf half below the limit. PR #27395 
does this. On that branch a bare marker with `OFFSET n LIMIT m` returns rows 
instead of the error, for both `file_row_index()` and `input_file_name()`.
   - Split out of #27131 at the reviewer's request; the measurements there came 
from the same runs.
   - Tested on `main` at `1841fb6ec`; line numbers are from that commit.
   
   ### Willingness to contribute
   
   I plan to contribute a fix for this bug independently (PR #27395).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to