discord9 opened a new issue, #23799:
URL: https://github.com/apache/datafusion/issues/23799
## Describe the bug
`LimitPushdown` can treat an operator-level `fetch` on a multi-partition
plan as
if it satisfied a global `LIMIT`.
For operators that do not combine partitions, `ExecutionPlan::fetch()` limits
each output partition independently. Removing the remaining global
enforcement
can therefore return up to `fetch * partition_count` rows. Related state
transitions can also place `GlobalLimitExec` directly above multiple
partitions
for OFFSET-only plans or combine a per-partition `LocalLimitExec` fetch with
a
pending global fetch.
## To Reproduce
Optimize a physical plan shaped like:
```text
GlobalLimitExec: skip=0, fetch=5
TestScan: partitions=2, supports_fetch=true
```
The current rule produces:
```text
TestScan: partitions=2, fetch=5
```
Each partition may emit five rows, so the plan can return ten rows even
though
the query has a global limit of five.
The same scope mismatch appears when the child already has a smaller
per-partition fetch, with OFFSET-only, and when a pending global limit
reaches a
multi-partition `LocalLimitExec`.
## Expected behavior
A fetch on a multi-partition operator should remain only an early-stop hint.
`LimitPushdown` should retain or create a single-partition enforcement
boundary
(`CoalescePartitionsExec` or `SortPreservingMergeExec`) before considering
the
global requirement satisfied. Global and per-partition local fetches should
not
be combined before that boundary.
## Additional context
The standard physical optimizer pipeline often inserts a partition-combining
boundary before `GlobalLimitExec`, which can hide this problem. It is
observable
when optimizing physical extension/distributed plan shapes and when the rule
itself removes or moves limit nodes. The fix can preserve per-partition
fetches
for early termination while retaining the global cap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
