wiedld commented on PR #13651: URL: https://github.com/apache/datafusion/pull/13651#issuecomment-2560616923
Fixed the performance regression. It wasn't where we thought it was. The problems was a recursive check (down the LP) of the check_fields within the `assert_unique_field_names()`. I've removed the recursive nature of this check. <details><summary>Final numbers, no regression</summary> ``` $ critcmp main invariants__no_recursive_namecheck__schema_check_per_optimizer_pass group invariants__no_recursive_namecheck__schema_check_per_optimizer_pass main ----- ------------------------------------------------------------------- ---- logical_aggregate_with_join 1.00 987.0±5.66µs ? ?/sec 1.02 1007.8±10.29µs ? ?/sec logical_select_all_from_1000 1.04 5.0±0.04ms ? ?/sec 1.00 4.8±0.04ms ? ?/sec logical_select_one_from_700 1.00 739.4±5.38µs ? ?/sec 1.01 743.5±5.70µs ? ?/sec logical_trivial_join_high_numbered_columns 1.00 703.2±5.00µs ? ?/sec 1.02 718.2±4.64µs ? ?/sec logical_trivial_join_low_numbered_columns 1.00 685.1±5.28µs ? ?/sec 1.02 699.6±3.40µs ? ?/sec physical_intersection 1.00 1726.0±10.72µs ? ?/sec 1.00 1726.6±15.43µs ? ?/sec physical_join_consider_sort 1.00 2.4±0.02ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec physical_join_distinct 1.00 677.2±5.94µs ? ?/sec 1.02 689.2±7.02µs ? ?/sec physical_many_self_joins 1.00 13.6±0.07ms ? ?/sec 1.02 13.9±0.07ms ? ?/sec physical_plan_clickbench_all 1.00 181.7±0.39ms ? ?/sec 1.00 181.5±0.36ms ? ?/sec physical_plan_clickbench_q1 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec physical_plan_clickbench_q10 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec physical_plan_clickbench_q11 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec physical_plan_clickbench_q12 1.00 3.6±0.01ms ? ?/sec 1.01 3.7±0.22ms ? ?/sec physical_plan_clickbench_q13 1.00 3.3±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec physical_plan_clickbench_q14 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec physical_plan_clickbench_q15 1.00 3.4±0.01ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec physical_plan_clickbench_q16 1.00 3.0±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec physical_plan_clickbench_q17 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec physical_plan_clickbench_q18 1.01 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q19 1.00 3.5±0.01ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec physical_plan_clickbench_q2 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q20 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec physical_plan_clickbench_q21 1.01 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q22 1.00 3.6±0.01ms ? ?/sec 1.01 3.6±0.01ms ? ?/sec physical_plan_clickbench_q23 1.00 4.0±0.01ms ? ?/sec 1.00 4.0±0.02ms ? ?/sec physical_plan_clickbench_q24 1.01 4.8±0.02ms ? ?/sec 1.00 4.7±0.01ms ? ?/sec physical_plan_clickbench_q25 1.01 3.2±0.02ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec physical_plan_clickbench_q26 1.01 2.9±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q27 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec physical_plan_clickbench_q28 1.00 3.7±0.01ms ? ?/sec 1.00 3.7±0.02ms ? ?/sec physical_plan_clickbench_q29 1.00 4.7±0.02ms ? ?/sec 1.00 4.7±0.01ms ? ?/sec physical_plan_clickbench_q3 1.01 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q30 1.01 15.5±0.41ms ? ?/sec 1.00 15.3±0.04ms ? ?/sec physical_plan_clickbench_q31 1.00 3.8±0.02ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec physical_plan_clickbench_q32 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec physical_plan_clickbench_q33 1.00 3.4±0.01ms ? ?/sec 1.00 3.4±0.01ms ? ?/sec physical_plan_clickbench_q34 1.00 3.1±0.01ms ? ?/sec 1.00 3.1±0.01ms ? ?/sec physical_plan_clickbench_q35 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec physical_plan_clickbench_q36 1.00 4.2±0.01ms ? ?/sec 1.00 4.2±0.02ms ? ?/sec physical_plan_clickbench_q37 1.00 4.3±0.01ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec physical_plan_clickbench_q38 1.00 4.3±0.02ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec physical_plan_clickbench_q39 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec physical_plan_clickbench_q4 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec physical_plan_clickbench_q40 1.00 4.3±0.01ms ? ?/sec 1.00 4.3±0.02ms ? ?/sec physical_plan_clickbench_q41 1.00 4.1±0.01ms ? ?/sec 1.00 4.1±0.02ms ? ?/sec physical_plan_clickbench_q42 1.00 4.0±0.02ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec physical_plan_clickbench_q43 1.00 4.0±0.01ms ? ?/sec 1.00 4.0±0.01ms ? ?/sec physical_plan_clickbench_q44 1.00 2.7±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec physical_plan_clickbench_q45 1.00 2.8±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec physical_plan_clickbench_q46 1.00 3.2±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec physical_plan_clickbench_q47 1.00 3.8±0.01ms ? ?/sec 1.00 3.8±0.01ms ? ?/sec physical_plan_clickbench_q48 1.00 4.3±0.01ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec physical_plan_clickbench_q49 1.00 4.5±0.02ms ? ?/sec 1.00 4.5±0.01ms ? ?/sec physical_plan_clickbench_q5 1.00 2.8±0.01ms ? ?/sec 1.00 2.7±0.01ms ? ?/sec physical_plan_clickbench_q6 1.00 2.8±0.01ms ? ?/sec 1.00 2.8±0.01ms ? ?/sec physical_plan_clickbench_q7 1.01 3.3±0.01ms ? ?/sec 1.00 3.3±0.01ms ? ?/sec physical_plan_clickbench_q8 1.00 3.0±0.01ms ? ?/sec 1.00 3.0±0.01ms ? ?/sec physical_plan_clickbench_q9 1.00 3.2±0.02ms ? ?/sec 1.00 3.3±0.02ms ? ?/sec physical_plan_tpcds_all 1.00 1160.9±2.22ms ? ?/sec 1.00 1161.8±5.74ms ? ?/sec physical_plan_tpch_all 1.00 72.5±0.23ms ? ?/sec 1.00 72.7±0.20ms ? ?/sec physical_plan_tpch_q1 1.00 2.6±0.01ms ? ?/sec 1.00 2.6±0.01ms ? ?/sec physical_plan_tpch_q10 1.00 3.5±0.01ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec physical_plan_tpch_q11 1.00 3.1±0.01ms ? ?/sec 1.01 3.2±0.17ms ? ?/sec physical_plan_tpch_q12 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec physical_plan_tpch_q13 1.00 1832.8±8.27µs ? ?/sec 1.00 1832.2±7.35µs ? ?/sec physical_plan_tpch_q14 1.00 2.2±0.01ms ? ?/sec 1.00 2.2±0.01ms ? ?/sec physical_plan_tpch_q16 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec physical_plan_tpch_q17 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec physical_plan_tpch_q18 1.00 3.2±0.01ms ? ?/sec 1.00 3.2±0.01ms ? ?/sec physical_plan_tpch_q19 1.00 5.3±0.01ms ? ?/sec 1.00 5.3±0.02ms ? ?/sec physical_plan_tpch_q2 1.01 6.4±0.03ms ? ?/sec 1.00 6.3±0.02ms ? ?/sec physical_plan_tpch_q20 1.00 3.9±0.01ms ? ?/sec 1.00 3.8±0.02ms ? ?/sec physical_plan_tpch_q21 1.00 5.1±0.01ms ? ?/sec 1.00 5.1±0.02ms ? ?/sec physical_plan_tpch_q22 1.00 2.9±0.01ms ? ?/sec 1.00 2.9±0.01ms ? ?/sec physical_plan_tpch_q3 1.00 2.5±0.01ms ? ?/sec 1.00 2.5±0.01ms ? ?/sec physical_plan_tpch_q4 1.00 1994.6±9.01µs ? ?/sec 1.00 1993.7±9.04µs ? ?/sec physical_plan_tpch_q5 1.01 3.6±0.02ms ? ?/sec 1.00 3.5±0.01ms ? ?/sec physical_plan_tpch_q6 1.00 1350.1±7.99µs ? ?/sec 1.00 1349.8±7.07µs ? ?/sec physical_plan_tpch_q7 1.01 4.7±0.02ms ? ?/sec 1.00 4.7±0.01ms ? ?/sec physical_plan_tpch_q8 1.01 5.6±0.03ms ? ?/sec 1.00 5.6±0.02ms ? ?/sec physical_plan_tpch_q9 1.00 4.3±0.02ms ? ?/sec 1.00 4.3±0.01ms ? ?/sec physical_select_aggregates_from_200 1.00 24.3±0.09ms ? ?/sec 1.00 24.3±0.05ms ? ?/sec physical_select_all_from_1000 1.01 43.2±0.35ms ? ?/sec 1.00 42.6±0.41ms ? ?/sec physical_select_one_from_700 1.00 2.5±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec physical_theta_join_consider_sort 1.00 2.7±0.02ms ? ?/sec 1.02 2.8±0.01ms ? ?/sec physical_unnest_to_join 1.00 2.4±0.01ms ? ?/sec 1.02 2.5±0.01ms ? ?/sec with_param_values_many_columns 1.00 183.9±0.62µs ? ?/sec 1.00 183.2±0.56µs ? ?/sec ``` </details> The release vs debug mode only has a single difference in the checks. The debug mode will run a full `InvariantLevel::Executable check` per each optimizer pass (instead of after all pass). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
