cool9850311 commented on issue #27457: URL: https://github.com/apache/datafusion/issues/27457#issuecomment-6953280916
I have a fix for this, in #27532 as a draft, and one question: which of two guards would you want? They are the two commits there. **(A)** `apply_limit` sets the fetch only on a `SortExec` with one output partition — what this issue suggests. **(B)** It sets the fetch on a multi-partition `SortExec` as well, but only when the limit came from a `SortPreservingMergeExec` whose ordering covers the window's `PARTITION BY`. Both fix every wrong-result shape I can find; (B) just changes fewer plans. The difference shows up on plans that are already sound: when the merge above the window orders the window's partitions apart, a row the window computed a wrong value for lost to `fetch` rows that all reach that merge before it, so it is past the merge's own fetch and never emitted. `datafusion/core/benches/topk_repartition.rs` is one of those plans. It runs with `target_partitions = 4`, so the `SortExec` below the window has four output partitions: (A) takes its fetch away and it becomes a full sort, (B) leaves the fetch in place. | `LIMIT` | `main` | with (A) | with (B) | |---|---|---|---| | 10 | 844 µs | **13.2 ms** | 858 µs | | 1,000 | 3.48 ms | **13.0 ms** | 3.49 ms | | 10,000 | 28.3 ms | 13.6 ms | 28.1 ms | | 100,000 | 123 ms | 30.6 ms | 127 ms | Medians of the `enabled` arm (the default config), `release-nonlto`, 10 samples, 3 s measurement. Each version measured in its own run with the machine otherwise idle; runs taken back to back under sustained load came out 10-30% high, so I discarded those. (A) is 16x slower at `LIMIT 10` and 3.7x at `LIMIT 1,000`, and 2.1x and 4.0x *faster* at the two larger limits — a TopK that keeps a large fraction of its input stops paying off at all, with the crossover near `LIMIT 4,000` on this data. The wrong results, measured on `120049489` and on each commit, 20 runs each: | shape | `main` | with (A) | with (B) | |---|---|---|---| | bare `LIMIT`, `TopKRepartition` on (this issue) | 20/20 wrong | 0/20 | 0/20 | | bare `LIMIT`, `TopKRepartition` off (this issue) | 20/20 wrong | 0/20 | 0/20 | | `PARTITION BY a, b` under `ORDER BY a` | 20/20 wrong | 0/20 | 0/20 | | the same, with a constant leading the `ORDER BY` | 20/20 wrong | 0/20 | 0/20 | The bottom two are shapes I hit while testing; each returns a row whose `row_number()` is 1 where the full result says 2. Reproducers for all four are in `window_limits.slt`. So: (A) now with that regression and (B) as a follow-up, or (B) straight away? Dropping the second commit gives (A). Either is fine by me — I just did not want to pick the larger change for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
