zhuqi-lucas commented on issue #23263: URL: https://github.com/apache/datafusion/issues/23263#issuecomment-4900468095
Ran the no-WHERE variant too: ```sql SELECT * FROM hits ORDER BY "EventTime" LIMIT 10; ``` ``` default: 409 ms avg (569 / 342 / 316) --pushdown: 279 ms avg (305 / 270 / 261) ``` You were right — [#22450](https://github.com/apache/datafusion/pull/22450) does most of the heavy lifting here (default is already 409ms). `pushdown_filters=true` adds a modest ~30% on top, because the runtime `EventTime > threshold` from TopK also gets applied as a `RowFilter` inside surviving row groups. On the "why not just default it to `true`" — there's an active EPIC tracking exactly this: [#20324](https://github.com/apache/datafusion/issues/20324) *"Fix performance regressions when enabling parquet filter pushdown (late materialization)"*. Q23 gets ~9x faster (alamb's Feb 2025 numbers), but **~18 other ClickBench queries regress by 5-55%** (Q2, Q6, Q7, Q10, Q11, Q12, Q13, Q14, Q19, Q24, Q25, Q26, Q27, Q28, Q37, Q40, Q41, Q42). Root cause is typically low-selectivity filters where the `RowFilter` bookkeeping outweighs the wide-decode savings. [#20325](https://github.com/apache/datafusion/issues/20325) tracks Q10 specifically. Also see [#22883](https://github.com/apache/datafusion/issues/22883) *"[EPIC] Adaptive predicate evaluation"* for a broader direction. So the trade-off is real — we'd want to close the regressions first, or make the pushdown decision adaptive per-filter (based on estimated selectivity). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
