Hi Everyone, It's been a while when I sent my email about introducing an automatic benchmark suite I had been working: https://externals.io/message/116323#116323 Now I would like to share an important milestone of this project:
After doing lots of improvements and fixes throughout the year, the suite now reached the required maturity level to be able to use it within PHP's nightly CI jobs (experimentally for now). Previously, consistency and reliability issues, as well as the lack of financial support prevented us from doing so. As these issues were addressed by now, we will be able to track the performance changes of PHP, starting with 8.5. The initial results are very promising for real-word app based tests (Laravel, Symfony, Wordpress), indicating that the results are consistent and seemingly reliable. For example run to run variation of the median absolute results typically don't exceed 0.001 sec. When benchmarking a php-src commit against itself, the relative difference for each real-word test varied between 0.01% and 0.06% (while 0.0% would have been the best possible value). Unfortunately, unlike real-word apps, micro benchmarks (bench.php and micro_bench.php) sometimes give inconsistent results, so there is still room for improvements. Besides making micro benchmarks more reliable, future plans include adding visualization, and coming up with some notification system which tells us if there is a notable performance regression or progression. For anyone interested in the project and the results, the benchmark suite is available at https://github.com/kocsismate/php-version-benchmarks, while the nightly results are committed into https://github.com/php/real-time-benchmark-data in both human friendly (markdown) and in machine processable format (tsv). Additionally, a more detailed explanation about the benchmark's methodology can be found at https://github.com/php/php-src/pull/15696. Regards, Máté
