cool9850311 commented on PR #27395:
URL: https://github.com/apache/datafusion/pull/27395#issuecomment-6950425409

   > @cool9850311 thanks for putting those two reasons back. alamb's last 
review asks for the symptom rather than the internals, and the PR description 
still opens with the rule-by-rule walk-through, line numbers and mutation 
tables (about 10k characters). Could you cut it down to the template's 
sections, along the lines below? I checked both issues' repros: on this PR's 
base `SELECT file_row_index() FROM small OFFSET 1 LIMIT 2` errors and `s['a']` 
under `OFFSET 5 LIMIT 3` scans `projection=[s]`. On this branch the first 
returns `1, 2` and the scan reads `projection=[get_field(s@0, a) ...]`.
   > 
   > Suggested shorter PR description
   > ```
   > ## Which issue does this PR close?
   > 
   > Closes #27494.
   > Closes #27495.
   > 
   > ## Rationale for this change
   > 
   > With `OFFSET n LIMIT m` on a listing table, `file_row_index()` and 
`input_file_name()` fail, and `s['a']` makes the scan read the whole struct:
   > 
   > ```sql
   > SELECT file_row_index() FROM small OFFSET 1 LIMIT 2;
   > -- Execution error: file_row_index() is source dependent and cannot be 
evaluated directly
   > ```
   > 
   > The skip pushed into the scan puts a `GlobalLimitExec` (and, with several 
files, a `CoalescePartitionsExec`) between the scan and the projection holding 
these expressions, and no projection moves below a `GlobalLimitExec`.
   > 
   > ## What changes are included in this PR?
   > 
   > The leaf-projection split that #27152 added for a round-robin 
`RepartitionExec` now also moves past `GlobalLimitExec` and 
`CoalescePartitionsExec`, including several stacked. Only the leaf expressions 
move below the limit; computed ones stay above it, so the rows the `OFFSET` 
skips don't evaluate them.
   > 
   > ## Are these changes tested?
   > 
   > Yes: value and `EXPLAIN` queries in `file_row_index.slt` and 
`projection_pushdown.slt`, and unit tests in 
`core/tests/physical_optimizer/projection_pushdown.rs` showing each input and 
output plan.
   > 
   > ## Are there any user-facing changes?
   > 
   > These queries now return rows instead of an error, and `s['a']` under 
`OFFSET` reads only that field. No API change.
   > ```
   
   Done.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to