Hello Internals, I'm writing this email because lately, I've been working on an automatic benchmarking framework for PHP, and I'd like to share some news regarding it. The initial implementation was sponsored by Craig Francis, and it was used for the evaluation of the performance aspects of the is_literal() RFC. Since then, I fixed numerous issues and implemented many new features, so with the very close advent of PHP 8.1, the time has come to unveil it.
I'm sure that many of you still remember Intel's automatic benchmarks from a couple of years ago (e.g. https://externals.io/message/89843#89843). I loved these emails, so this project served as a great inspiration for me to start working on something similar. I have to admit though that I won't be able to replicate their extremely advanced setup. My main goal was to develop a framework ( https://github.com/kocsismate/php-version-benchmarks) which is: - fully automatic so that it can be easily run regularly, and the benchmarks are reproducible - it's possible to try it out locally via Docker, but it can be run in the cloud (currently, only AWS is supported), and the instance type is configurable - supports different CPU platforms (X86-64 and ARM64), and advanced options whether turbo boost/hyper threading/deeper CPU C-states are enabled is configurable - supports any version of PHP since PHP 7.4, including any git branches or commits - supports the most important PHP configurations, including whether OPcache/JIT/preloading is enabled - supports multiple tests: currently, the micro benchmarks bundled with php-src, as well as the Symfony and the Laravel demo sites as "real-life" tests are included with some customizability (number of warmups, iterations number, number of requests) - results are measured via using PHP-CGI which is a lightweight web server with very a small overhead The most current benchmark is available at https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32% faster than PHP 7.4 in real-life tests, and a few percent faster when it comes to micro benchmarks. The benchmark was performed on an ARM64 instance because this platform provided much more stable results than X86-64-based ones did, mainly due to their fixed CPU frequency. Unfortunately, the benchmark is not yet suitable for detecting minor performance changes between commits as the run-to-run variation of the results is a bit too high; in some cases, it can be up to 1-2%. I hope that this problem can be mitigated to an acceptable level in the future, but most probably I'll need some help to achieve this. So any help is appreciated! Furthermore, I'd have some questions regarding this project: - Would you welcome automatic emails on internals (or on a dedicated mailing list), just like how Intel did it? I'm not sure about the frequency, but in my opinion, 1-2/month would be a sensible one, given there is interest in it. - Does anyone know how to get some sponsorship from AWS (or from some other cloud provider at last resort)? I already tried to apply for AWS Activate's free credits, but my application was rejected due to different reasons (normally, this program is available for startups). I have some hope that running the benchmark on a dedicated instance would decrease variation of the results, but enabling this feature is relatively expensive compared to other costs, so I'd be very happy if I wouldn't have to sponsor all of this, when I already dedicate a great deal of my time to the cause. - As always, I'm happy to receive feedback and improvements (ideally along with an implementation :) ). For example, currently I've started working on some visualization and chart support, but frontend stuff is not my area of expertise, so I'm progressing a bit slower than normally. Also, the statistical foundations could be reviewed, and possibly improved. Regards: Máté
