mohitgurav20 commented on PR #25719: URL: https://github.com/apache/datafusion/pull/25719#issuecomment-6370374701
Hi @asolimando, I ran `EXPLAIN` for TPC-DS SF1 q4, q11, q47, and q67 on both `main` and this branch, and the generated plans (both optimized logical plans and physical execution plans) are 100% byte-for-byte identical across all four queries. ### Why the plans do not differ: - The fallback heuristic (`compute_fallback_selectivity`) only changes the selectivity estimate when interval analysis fails and an equality conjunct has a bare column with known, non-zero NDV (`column_ndv`). - In **q4** and **q11**, the filters evaluate over CTEs formed by `UNION ALL` aggregations. Because derived aggregate projections do not carry column distinct-count statistics, `column_ndv` returns `None` and falls back to the exact default 20% selectivity used on `main`. - In **q47**, the outer filters are on window function outputs (`avg_monthly_sales > 0`, `case ... > 0.1`) alongside `d_year = 2000` (handled by interval analysis, not the fallback). - In **q67**, the filter is `rk <= 100` on a window rank function (range comparison, not equality). Since the physical plans, join types, and join orderings are completely identical, the ~2% run-to-run delta (averaging ~2 ms per query: 88.6 ms vs 90.6 ms) and the peak memory shift in the 99-query suite appear to be execution-time batch buffering and runner noise on the multi-tenant GKE instance rather than an optimizer plan regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
