Given the recent benchmarks from Phoronix (https://www.phoronix.com/scan.php?page=article&item=fedora-frame-pointer&num=1) on the proposal that showed some surprising results, we went and tried to reproduce some of the benchmarks to make sure they were actually making sense.
The first one we looked at is the redis benchmark from https://www.phoronix.com/scan.php?page=article&item=fedora-frame-pointer&num=5. We were unable to reproduce the results from the Phoronix article. Redis GET: https://user-images.githubusercontent.com/9395011/176536797-7424d40f-7140-46f8-89d3-7b555aa4cd13.png Redis SET: https://user-images.githubusercontent.com/9395011/176536624-eeb5f85c-a63b-4987-8b3b-2b3607be0cf8.png Instead, we only saw differences from 0%-2% between Redis compiled with frame pointers and Redis compiled without frame pointers. These benchmarks were done using the phoronix-test-suite in exactly the same way as documented in the phoronix article. The other one we've looked at is the Botan AES-256 benchmark (https://www.phoronix.com/scan.php?page=article&item=fedora-frame-pointer&num=2). Initially, we were able to reproduce the results of this benchmark when setting CXXFLAGS="-fno-omit-frame-pointer". However, what we found here, is that due to the way Botan's custom build system works, when the CXXFLAGS environment variable is set to enable -fno-omit-frame-pointer, the botan binary is built in debug mode without optimizations whereas when CXXFLAGS is unset, the botan binary is build in release mode (-O3). This explains the huge difference in performance in the botan AES-256 benchmark. When making sure both binaries are built in release mode by setting CXXFLAGS="-O2" and CXXFLAGS="-O2 -fno-omit-frame-pointer" respectively, we get the following results : Without frame pointers: AES-256 encrypt buffer size 1024 bytes: 5410.085 MiB/sec 0.42 cycles/byte (2705.04 MiB in 500.00 ms) AES-256 decrypt buffer size 1024 bytes: 5407.610 MiB/sec 0.42 cycles/byte (2703.81 MiB in 500.00 ms) With frame pointers: AES-256 encrypt buffer size 1024 bytes: 5359.241 MiB/sec 0.42 cycles/byte (2679.62 MiB in 500.00 ms) AES-256 decrypt buffer size 1024 bytes: 5404.226 MiB/sec 0.42 cycles/byte (2702.11 MiB in 500.00 ms) Which shows a smaller than 1% slowdown between the binary built with frame pointers and the binary built without frame pointers. Supposedly, the Phoronix benchmark was also built with "-O2" in both configurations but given that we saw very similar results to what was in the phoronix benchmark result when building Botan in debug mode, we assume that's what happened with the AES Botan benchmark. We haven't yet dived deeper into the other benchmarks, but we expect that the benchmarks showing significant differences might suffer from similar issues, where the huge differences are not caused by the inclusion of frame pointers, but other unrelated issues such as the botan case where setting CXXFLAGS causes binaries to be built in debug mode unless an explicit optimization mode is set. These benchmarks were done on an Amazon EC2 instance running Fedora 36 Cloud edition. The full details as reported by phoronix-test-suite can be found here: https://user-images.githubusercontent.com/9395011/176538700-c82974fa-fbb5-4146-be96-d6db1ce7dfb0.png _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
