SubhamSinghal opened a new pull request, #27197: URL: https://github.com/apache/datafusion/pull/27197
## Which issue does this PR close? - Closes #24608. ## Rationale for this change `enable_window_topn` has been off since #21479 because of high-cardinality regressions against the sort plan. The work items in #24608 are now done: - #23096, - #25730, - #25968, - https://github.com/apache/datafusion/pull/26133 The h2o window benchmark's Top-N queries (`benchmarks/queries/h2o/window.sql`, Q13–Q29, small dataset, `J1_1e7_1e7_NA.parquet`) are now faster with the flag on in every case, up to 100K partitions and including the heavy-ties variants. `dfbench h2o --iterations 5`, median, `DATAFUSION_OPTIMIZER_ENABLE_WINDOW_TOPN=false` for the baseline, same binary, Apple M4 Max: | Query | function, partitions | off (ms) | on (ms) | change | |---|---|---:|---:|---| | Q13 | ROW_NUMBER, id2 | 127.4 | 25.8 | 4.93x faster | | Q14 | ROW_NUMBER, 100 | 190.8 | 35.9 | 5.32x faster | | Q15 | ROW_NUMBER, 1K | 173.6 | 35.7 | 4.86x faster | | Q16 | ROW_NUMBER, 10K | 149.7 | 33.0 | 4.54x faster | | Q17 | ROW_NUMBER, 100K | 151.1 | 49.1 | 3.08x faster | | Q18 | RANK, 100 | 226.3 | 28.8 | 7.86x faster | | Q19 | RANK, 1K | 186.0 | 29.6 | 6.28x faster | | Q20 | RANK, 1K, heavy ties | 209.8 | 57.1 | 3.67x faster | | Q21 | RANK, id2 | 174.2 | 25.4 | 6.87x faster | | Q22 | RANK, id2, heavy ties | 190.5 | 37.2 | 5.13x faster | | Q23 | RANK, 100K | 188.1 | 63.5 | 2.96x faster | | Q24 | DENSE_RANK, 100 | 248.4 | 38.6 | 6.43x faster | | Q25 | DENSE_RANK, 1K | 199.6 | 36.9 | 5.40x faster | | Q26 | DENSE_RANK, 1K, heavy ties | 232.2 | 43.3 | 5.37x faster | | Q27 | DENSE_RANK, id2 | 211.3 | 45.7 | 4.63x faster | | Q28 | DENSE_RANK, id2, heavy ties | 208.7 | 53.6 | 3.89x faster | | Q29 | DENSE_RANK, 100K | 205.8 | 123.3 | 1.67x faster | ## What changes are included in this PR? - `enable_window_topn` defaults to `true`; its doc comment now lists ROW_NUMBER, RANK and DENSE_RANK and drops the high-cardinality warning. - `window_topn.slt` restores the setting with `RESET` instead of `SET ... = false`. - `configs.md` and `information_schema.slt` regenerated. - The h2o `window_sorted` settings comment no longer says the flag defaults to false (it still sets it explicitly). ## What is the testing strategy for this PR? Existing coverage. With the flag on by default, the whole sqllogictest suite and `core_integration` pass with only the `information_schema.slt` default change; `qualify.slt`'s `rank <= 3` / `dense_rank <= 2` cases now run through `PartitionedTopKExec` and return the same results. `window_topn.slt` still covers the disabled path explicitly. ## Are there any user-facing changes? Yes. Top-K-per-partition queries (`rn <= K`, `rn < K` over ROW_NUMBER / RANK / DENSE_RANK) plan with `PartitionedTopKExec` instead of a full sort, so `EXPLAIN` output changes; results don't. Set `datafusion.optimizer.enable_window_topn = false` to get the old plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
