Hey Máté, On Fri, Nov 12, 2021 at 11:19 AM Máté Kocsis <kocsismat...@gmail.com> wrote:
> Hello Internals, > > I'm writing this email because lately, I've been working on an automatic > benchmarking framework for PHP, and I'd like to share some news regarding > it. The initial implementation > was sponsored by Craig Francis, and it was used for the evaluation of the > performance aspects of the is_literal() RFC. Since then, I fixed numerous > issues and implemented many new features, so with the very close advent of > PHP 8.1, the time has come to unveil it. > > I'm sure that many of you still remember Intel's automatic benchmarks from > a couple of years ago (e.g. https://externals.io/message/89843#89843). I > loved these emails, so this project served as a great inspiration for me > to start working on something similar. I have to admit > though that I won't be able to replicate their extremely advanced setup. > > My main goal was to develop a framework ( > https://github.com/kocsismate/php-version-benchmarks) which is: > - fully automatic so that it can be easily run regularly, and the > benchmarks are reproducible > - it's possible to try it out locally via Docker, but it can be run in the > cloud (currently, only AWS is > supported), and the instance type is configurable > - supports different CPU platforms (X86-64 and ARM64), and advanced options > whether > turbo boost/hyper threading/deeper CPU C-states are enabled is configurable > - supports any version of PHP since PHP 7.4, including any git branches or > commits > - supports the most important PHP configurations, including whether > OPcache/JIT/preloading > is enabled > - supports multiple tests: currently, the micro benchmarks bundled with > php-src, as well as the > Symfony and the Laravel demo sites as "real-life" tests are included with > some customizability (number of warmups, iterations number, number of > requests) > - results are measured via using PHP-CGI which is a lightweight web server > with very a small overhead > > The most current benchmark is available at > > https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md > . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32% > faster than PHP 7.4 in real-life tests, and a few percent faster when it > comes to micro benchmarks. The benchmark was performed on an ARM64 instance > because this platform provided much more stable results than X86-64-based > ones did, mainly due to their fixed CPU frequency. > > Unfortunately, the benchmark is not yet suitable for detecting minor > performance changes between commits as the run-to-run variation of the > results is a bit too high; in some cases, it can be up to 1-2%. I hope that > this problem can be mitigated to an acceptable level in the future, but > most probably I'll need some help to achieve this. So any help is > appreciated! > > Furthermore, I'd have some questions regarding this project: > - Would you welcome automatic emails on internals (or on a dedicated > mailing list), just like how > Intel did it? I'm not sure about the frequency, but in my opinion, > 1-2/month would be a sensible one, given there is interest in it. > - Does anyone know how to get some sponsorship from AWS (or from some other > cloud > provider at last resort)? I already tried to apply for AWS Activate's free > credits, but my application was rejected due to different reasons > (normally, this program is available for startups). I have some hope that > running the benchmark on a dedicated instance would decrease variation of > the results, but enabling this feature is relatively expensive compared to > other costs, so I'd be very happy if I wouldn't have to sponsor all of > this, when I already dedicate a great deal of my time to the cause. > - As always, I'm happy to receive feedback and improvements (ideally along > with an implementation :) ). For example, currently I've started working on > some visualization and chart support, but frontend stuff is not my area of > expertise, so I'm progressing a bit slower than normally. Also, the > statistical foundations could be reviewed, and possibly improved. > > Regards: > Máté > Nice results, and nice project! Overall against having it on the ML: I set up a filter to ignore those pesky emails by Intel. As much as I appreciate their effort, they were really noisy, and not really useful to the larger group. If your tool could export the results in machine format with something like an attached XSD or JSON-Schema, it would be possible to build UIs like https://arewefastyet.com/ , without the need to resort to a notification model. What's needed is to save them in a branch on the repo (or similar approach) for later consumption. Greets, Marco Pivetta http://twitter.com/Ocramius http://ocramius.github.com/ On Fri, Nov 12, 2021 at 11:19 AM Máté Kocsis <kocsismat...@gmail.com> wrote: > Hello Internals, > > I'm writing this email because lately, I've been working on an automatic > benchmarking framework for PHP, and I'd like to share some news regarding > it. The initial implementation > was sponsored by Craig Francis, and it was used for the evaluation of the > performance aspects of the is_literal() RFC. Since then, I fixed numerous > issues and implemented many new features, so with the very close advent of > PHP 8.1, the time has come to unveil it. > > I'm sure that many of you still remember Intel's automatic benchmarks from > a couple of years ago (e.g. https://externals.io/message/89843#89843). I > loved these emails, so this project served as a great inspiration for me > to start working on something similar. I have to admit > though that I won't be able to replicate their extremely advanced setup. > > My main goal was to develop a framework ( > https://github.com/kocsismate/php-version-benchmarks) which is: > - fully automatic so that it can be easily run regularly, and the > benchmarks are reproducible > - it's possible to try it out locally via Docker, but it can be run in the > cloud (currently, only AWS is > supported), and the instance type is configurable > - supports different CPU platforms (X86-64 and ARM64), and advanced options > whether > turbo boost/hyper threading/deeper CPU C-states are enabled is configurable > - supports any version of PHP since PHP 7.4, including any git branches or > commits > - supports the most important PHP configurations, including whether > OPcache/JIT/preloading > is enabled > - supports multiple tests: currently, the micro benchmarks bundled with > php-src, as well as the > Symfony and the Laravel demo sites as "real-life" tests are included with > some customizability (number of warmups, iterations number, number of > requests) > - results are measured via using PHP-CGI which is a lightweight web server > with very a small overhead > > The most current benchmark is available at > > https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md > . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32% > faster than PHP 7.4 in real-life tests, and a few percent faster when it > comes to micro benchmarks. The benchmark was performed on an ARM64 instance > because this platform provided much more stable results than X86-64-based > ones did, mainly due to their fixed CPU frequency. > > Unfortunately, the benchmark is not yet suitable for detecting minor > performance changes between commits as the run-to-run variation of the > results is a bit too high; in some cases, it can be up to 1-2%. I hope that > this problem can be mitigated to an acceptable level in the future, but > most probably I'll need some help to achieve this. So any help is > appreciated! > > Furthermore, I'd have some questions regarding this project: > - Would you welcome automatic emails on internals (or on a dedicated > mailing list), just like how > Intel did it? I'm not sure about the frequency, but in my opinion, > 1-2/month would be a sensible one, given there is interest in it. > - Does anyone know how to get some sponsorship from AWS (or from some other > cloud > provider at last resort)? I already tried to apply for AWS Activate's free > credits, but my application was rejected due to different reasons > (normally, this program is available for startups). I have some hope that > running the benchmark on a dedicated instance would decrease variation of > the results, but enabling this feature is relatively expensive compared to > other costs, so I'd be very happy if I wouldn't have to sponsor all of > this, when I already dedicate a great deal of my time to the cause. > - As always, I'm happy to receive feedback and improvements (ideally along > with an implementation :) ). For example, currently I've started working on > some visualization and chart support, but frontend stuff is not my area of > expertise, so I'm progressing a bit slower than normally. Also, the > statistical foundations could be reviewed, and possibly improved. > > Regards: > Máté >
