cool9850311 commented on issue #27457:
URL: https://github.com/apache/datafusion/issues/27457#issuecomment-6952365740

   @jayzhan211 
   I have a fix for this, and one question: which of two guards would you want?
   
   **(A)** Set the fetch only on a `SortExec` with one output partition — what 
the issue
   suggests.
   
   **(B)** Set it on a multi-partition `SortExec` too, but only when the limit 
came from a
   `SortPreservingMergeExec` whose ordering covers the window's `PARTITION BY`.
   
   Both fix every wrong-result shape I can find; (B) just changes fewer plans. 
The
   difference shows up on plans that are already sound: when the merge above 
the window
   orders the window's partitions apart, a row the window computed a wrong 
value for lost to
   `fetch` rows that all reach that merge before it, so it is past the merge's 
own fetch and
   never emitted.
   
   `datafusion/core/benches/topk_repartition.rs` is one of those plans. It runs 
with
   `target_partitions = 4`, so the `SortExec` below the window has four output 
partitions:
   (A) takes its fetch away and it becomes a full sort, (B) leaves the fetch in 
place.
   
   | `LIMIT` | `main` | with (A) | with (B) |
   |---|---|---|---|
   | 10 | 907 µs | **15.3 ms** | 836 µs |
   | 1,000 | 3.66 ms | **15.5 ms** | 3.56 ms |
   | 10,000 | 28.3 ms | 14.7 ms | 28.7 ms |
   | 100,000 | 128 ms | 32.3 ms | 123 ms |
   
   Medians, `release-nonlto`, 10 samples, 3 s measurement, default config. Two 
runs of one
   build differ by up to 13% at the two largest limits, so (B) is unchanged 
from `main`.
   Above roughly `LIMIT 4,000` (A) is *faster*, because a TopK that keeps a 
large fraction of
   its input stops paying off at all.
   
   The wrong results, which (A) and (B) fix alike:
   
   | shape | on `main` | with either guard |
   |---|---|---|
   | bare `LIMIT`, `TopKRepartition` on (this issue) | 20/20 wrong | 0/20 |
   | bare `LIMIT`, `TopKRepartition` off (this issue) | 20/20 wrong | 0/20 |
   | `PARTITION BY a, b` under `ORDER BY a` | 20/20 wrong | 0/20 |
   | the same, with a constant leading the `ORDER BY` | 19/20 wrong | 0/20 |
   
   The bottom two are shapes I hit while testing. Each returns a row whose 
`row_number()` is
   1 where the full result says 2. Reproducers will be in the PR either way.
   
   So: (A) now with that regression and (B) as a follow-up, or (B) straight 
away? Either is
   fine by me — I just did not want to pick the larger change for you.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to