FTR, by default R on macOS uses system clang, not homebrew; for Windows, it is gcc (unless you're adventurous like Uwe and can hack it to work with clang ;); and on Linux, CRAN checks both gcc and clang.
Is it unreasonable to check *both* gcc and clang when there's a potentially significant performance change? It sounded like recently there have been cases where a change makes a big speedup with one compiler but a slowdown with the other, so if we only test with one, we'll be degrading with the other over time. But if we test both (simplifying of course, since there are multiple versions, etc.), then at least we'd be more aware of what the tradeoff is. Neal On Mon, Jun 22, 2020 at 8:42 AM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > We should aim to improve the performance of the most widely used > *default* packages, which are python pip, python conda and R (all > platforms). AFAIK, both pip (manywheel) and conda use gcc on Linux by > default. R uses gcc on Linux and mingw (gcc) on Windows. I suppose > (haven't checked) that clang is used on OSX via brew. Thus, by > default, almost all users are going to use a gcc compiled version of > arrow on Linux. > > François > > On Mon, Jun 22, 2020 at 9:47 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > > Based on some of my performance work recently, I'm growing > > uncomfortable with using gcc as the performance baseline since the > > results can be significantly different (sometimes 3-4x or more on > > certain fast algorithms) from clang and MSVC. The perf results on > > https://github.com/apache/arrow/pull/7506 were really surprising -- > > some benchmarks that showed 2-5x performance improvement on both clang > > and MSVC shows small regressions (20-30%) with gcc. > > > > I don't think we need a hard-and-fast rule about whether to accept PRs > > based on benchmarks but there are a few guiding criteria: > > > > * How much binary size does the new code add? I think many of us would > > agree that a 20% performance increase on some algorithm might not be > > worth adding 500KB to libarrow.so > > * Is the code generally faster across the major compiler targets (gcc, > > clang, MSVC)? > > > > I think that using clang as a baseline for informational benchmarks > > would be good, but ultimately we need to be systematically collecting > > data on all the major compiilers. Some time ago I proposed building a > > Continuous Benchmarking framework > > (https://github.com/conbench/conbench/blob/master/doc/REQUIREMENTS.md) > > for use with Arrow (and outside of Arrow, too) so I hope that this > > will be able to help. > > > > - Wes > > > > On Mon, Jun 22, 2020 at 5:12 AM Yibo Cai <yibo....@arm.com> wrote: > > > > > > On 6/22/20 5:07 PM, Antoine Pitrou wrote: > > > > > > > > Le 22/06/2020 à 06:27, Micah Kornfield a écrit : > > > >> There has been significant effort recently trying to optimize our > C++ > > > >> code. One thing that seems to come up frequently is different > benchmark > > > >> results between GCC and Clang. Even different versions of the same > > > >> compiler can yield significantly different results on the same code. > > > >> > > > >> I would like to propose that we choose a specific compiler and > version on > > > >> Linux for evaluating performance related PRs. PRs would only be > accepted > > > >> if they improve the benchmarks under the selected version. > > > > > > > > Would this be a hard rule or just a guideline? There are many ways > in > > > > which benchmark numbers can be improved or deteriorated by a PR, and > in > > > > some cases that doesn't matter (benchmarks are not always realistic, > and > > > > they are not representative of every workload). > > > > > > > > > > I agree that microbenchmark is not always useful, focusing too much on > > > improving microbenchmark result gives me feeling of "overfit" (to some > > > specific microarchitecture, compiler, or use case). > > > > > > > Regards > > > > > > > > Antoine. > > > > >
