alamb commented on PR #15462: URL: https://github.com/apache/datafusion/pull/15462#issuecomment-2775475696
> > I looked for a binary expression and/or micro benchmark and could not find one: https://github.com/apache/datafusion/tree/c929a1cd133590e4944bc2c7900611220450335a/datafusion/physical-expr/benches > > Thank you for your suggestions and help. I also noticed this part of the code yesterday. I've had a cold and fever in the past few days, so I haven't tried the method mentioned by @Dandandan for benchmarking yet. I should be able to work on it today. I hope you feel better soon I have been running some benchmark numbers (below) and I have gotten some strange results. I definitely see this PR improves performance on the newly added clickbench extended benchmark (almost 3x faster!) However `clickbench_1` seems to show a slowdown compared to `clickbench_partitioned` which I don't understand. I will try and reproduce this locally <details><summary>Details</summary> <p> ``` -------------------- Benchmark clickbench_1.json -------------------- ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓ ┃ Query ┃ main_base ┃ add_short_circuit ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩ │ QQuery 0 │ 0.58ms │ 0.59ms │ no change │ │ QQuery 1 │ 70.93ms │ 72.44ms │ no change │ │ QQuery 2 │ 118.33ms │ 120.96ms │ no change │ │ QQuery 3 │ 124.52ms │ 130.17ms │ no change │ │ QQuery 4 │ 788.26ms │ 835.71ms │ 1.06x slower │ │ QQuery 5 │ 891.18ms │ 1033.71ms │ 1.16x slower │ │ QQuery 6 │ 68.02ms │ 66.47ms │ no change │ │ QQuery 7 │ 79.51ms │ 80.38ms │ no change │ │ QQuery 8 │ 951.66ms │ 962.46ms │ no change │ │ QQuery 9 │ 1242.83ms │ 1277.46ms │ no change │ │ QQuery 10 │ 305.01ms │ 312.25ms │ no change │ │ QQuery 11 │ 334.19ms │ 341.58ms │ no change │ │ QQuery 12 │ 941.89ms │ 1079.60ms │ 1.15x slower │ │ QQuery 13 │ 1341.24ms │ 1528.55ms │ 1.14x slower │ │ QQuery 14 │ 874.85ms │ 1057.59ms │ 1.21x slower │ │ QQuery 15 │ 1118.77ms │ 1134.54ms │ no change │ │ QQuery 16 │ 1803.30ms │ 1920.39ms │ 1.06x slower │ │ QQuery 17 │ 1669.97ms │ 1780.42ms │ 1.07x slower │ │ QQuery 18 │ 3269.95ms │ 3297.16ms │ no change │ │ QQuery 19 │ 120.62ms │ 116.02ms │ no change │ │ QQuery 20 │ 1222.20ms │ 1302.52ms │ 1.07x slower │ │ QQuery 21 │ 1471.42ms │ 1638.85ms │ 1.11x slower │ │ QQuery 22 │ 2740.23ms │ 4508.20ms │ 1.65x slower │ │ QQuery 23 │ 8733.62ms │ 10619.68ms │ 1.22x slower │ │ QQuery 24 │ 515.74ms │ 695.11ms │ 1.35x slower │ │ QQuery 25 │ 431.26ms │ 608.92ms │ 1.41x slower │ │ QQuery 26 │ 568.41ms │ 743.79ms │ 1.31x slower │ │ QQuery 27 │ 1865.82ms │ 1966.63ms │ 1.05x slower │ │ QQuery 28 │ 13400.82ms │ 13320.07ms │ no change │ │ QQuery 29 │ 574.24ms │ 590.87ms │ no change │ │ QQuery 30 │ 863.90ms │ 1059.33ms │ 1.23x slower │ │ QQuery 31 │ 940.63ms │ 1115.46ms │ 1.19x slower │ │ QQuery 32 │ 2790.62ms │ 2933.31ms │ 1.05x slower │ │ QQuery 33 │ 3534.40ms │ 3583.06ms │ no change │ │ QQuery 34 │ 3515.54ms │ 3592.40ms │ no change │ │ QQuery 35 │ 1327.76ms │ 1384.98ms │ no change │ │ QQuery 36 │ 257.33ms │ 269.35ms │ no change │ │ QQuery 37 │ 124.48ms │ 187.43ms │ 1.51x slower │ │ QQuery 38 │ 167.20ms │ 182.73ms │ 1.09x slower │ │ QQuery 39 │ 454.63ms │ 474.86ms │ no change │ │ QQuery 40 │ 82.41ms │ 83.28ms │ no change │ │ QQuery 41 │ 82.20ms │ 80.21ms │ no change │ │ QQuery 42 │ 78.13ms │ 87.76ms │ 1.12x slower │ └──────────────┴────────────┴───────────────────┴──────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ Total Time (main_base) │ 61858.61ms │ │ Total Time (add_short_circuit) │ 68177.23ms │ │ Average Time (main_base) │ 1438.57ms │ │ Average Time (add_short_circuit) │ 1585.52ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 21 │ │ Queries with No Change │ 22 │ └──────────────────────────────────┴────────────┘ -------------------- Benchmark clickbench_extended.json -------------------- ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ main_base ┃ add_short_circuit ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 0 │ 1914.62ms │ 1954.11ms │ no change │ │ QQuery 1 │ 757.11ms │ 760.32ms │ no change │ │ QQuery 2 │ 1478.24ms │ 1495.54ms │ no change │ │ QQuery 3 │ 700.15ms │ 705.43ms │ no change │ │ QQuery 4 │ 1489.20ms │ 1457.77ms │ no change │ │ QQuery 5 │ 17258.06ms │ 17048.63ms │ no change │ │ QQuery 6 │ 6872.16ms │ 2318.32ms │ +2.96x faster │ └──────────────┴────────────┴───────────────────┴───────────────┘ -------------------- Benchmark clickbench_partitioned.json -------------------- ┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓ ┃ Query ┃ main_base ┃ add_short_circuit ┃ Change ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩ │ QQuery 0 │ 2.45ms │ 2.71ms │ 1.11x slower │ │ QQuery 1 │ 36.96ms │ 38.85ms │ 1.05x slower │ │ QQuery 2 │ 93.10ms │ 98.35ms │ 1.06x slower │ │ QQuery 3 │ 99.41ms │ 102.08ms │ no change │ │ QQuery 4 │ 771.09ms │ 762.77ms │ no change │ │ QQuery 5 │ 880.06ms │ 878.60ms │ no change │ │ QQuery 6 │ 37.74ms │ 34.11ms │ +1.11x faster │ │ QQuery 7 │ 42.34ms │ 43.34ms │ no change │ │ QQuery 8 │ 961.07ms │ 940.11ms │ no change │ │ QQuery 9 │ 1212.05ms │ 1215.74ms │ no change │ │ QQuery 10 │ 272.28ms │ 280.09ms │ no change │ │ QQuery 11 │ 314.24ms │ 321.60ms │ no change │ │ QQuery 12 │ 947.28ms │ 945.83ms │ no change │ │ QQuery 13 │ 1412.66ms │ 1420.04ms │ no change │ │ QQuery 14 │ 883.62ms │ 889.96ms │ no change │ │ QQuery 15 │ 1066.12ms │ 1084.71ms │ no change │ │ QQuery 16 │ 1826.85ms │ 1763.33ms │ no change │ │ QQuery 17 │ 1654.90ms │ 1636.73ms │ no change │ │ QQuery 18 │ 3133.68ms │ 3105.68ms │ no change │ │ QQuery 19 │ 88.53ms │ 91.74ms │ no change │ │ QQuery 20 │ 1154.68ms │ 1136.04ms │ no change │ │ QQuery 21 │ 1387.95ms │ 1325.31ms │ no change │ │ QQuery 22 │ 2526.34ms │ 2333.56ms │ +1.08x faster │ │ QQuery 23 │ 8563.31ms │ 8668.93ms │ no change │ │ QQuery 24 │ 486.09ms │ 481.52ms │ no change │ │ QQuery 25 │ 410.40ms │ 407.97ms │ no change │ │ QQuery 26 │ 543.51ms │ 554.22ms │ no change │ │ QQuery 27 │ 1721.79ms │ 1685.69ms │ no change │ │ QQuery 28 │ 12803.31ms │ 12658.72ms │ no change │ │ QQuery 29 │ 526.02ms │ 537.58ms │ no change │ │ QQuery 30 │ 863.88ms │ 860.27ms │ no change │ │ QQuery 31 │ 907.18ms │ 909.85ms │ no change │ │ QQuery 32 │ 2749.24ms │ 2724.90ms │ no change │ │ QQuery 33 │ 3434.16ms │ 3391.96ms │ no change │ │ QQuery 34 │ 3406.33ms │ 3378.15ms │ no change │ │ QQuery 35 │ 1306.15ms │ 1301.79ms │ no change │ │ QQuery 36 │ 230.41ms │ 223.25ms │ no change │ │ QQuery 37 │ 89.36ms │ 88.77ms │ no change │ │ QQuery 38 │ 129.02ms │ 127.27ms │ no change │ │ QQuery 39 │ 418.02ms │ 408.13ms │ no change │ │ QQuery 40 │ 52.52ms │ 48.55ms │ +1.08x faster │ │ QQuery 41 │ 45.36ms │ 46.73ms │ no change │ │ QQuery 42 │ 55.92ms │ 55.00ms │ no change │ └──────────────┴────────────┴───────────────────┴───────────────┘ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓ ┃ Benchmark Summary ┃ ┃ ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩ │ Total Time (main_base) │ 59547.38ms │ │ Total Time (add_short_circuit) │ 59010.53ms │ │ Average Time (main_base) │ 1384.82ms │ │ Average Time (add_short_circuit) │ 1372.34ms │ │ Queries Faster │ 3 │ │ Queries Slower │ 3 │ │ Queries with No Change │ 37 │ └──────────────────────────────────┴────────────┘ ``` </p> </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
