There has been significant effort recently trying to optimize our C++ code. One thing that seems to come up frequently is different benchmark results between GCC and Clang. Even different versions of the same compiler can yield significantly different results on the same code.
I would like to propose that we choose a specific compiler and version on Linux for evaluating performance related PRs. PRs would only be accepted if they improve the benchmarks under the selected version. Other performance related PRs would still be acceptable if they improve benchmarks for a different compiler as long as they don't make the primary one worse and don't hurt maintainability of the code. I also don't think this should change our CI matrix in any way. There are other variables that potentially come into play for evaluating benchmarks (architecture, OS Version, etc) but if we need to limit these in a similar way, I think they should be handled as separate discussions. I'm not clear on the limitations that our distributions place on compilers, but ideally we'd pick a compiler that can produce most/all of our binary artifacts. Could someone more familiar with the release toolchain comment on good options to consider? Any other criteria we should consider? Other thoughts? Thanks, Micah
