Hi Even. Thanks, it sounds good. However I see a potential problem. I see that you use once "SetCacheMax". We should not forget about that in the future for sensible tests. The cache of gdal is usually a percentage of the total memory, that may change among the environments and time.
On Wed, 11 Oct 2023, 07:53 Laurențiu Nicola via gdal-dev, < gdal-dev@lists.osgeo.org> wrote: > Hi, > > No experience with pytest-benchmark, but I maintain an unrelated project > that runs some benchmarks on CI, and here are some things worth mentioning: > > - we store the results as a newline-delimited JSON file in a different > GitHub repository ( > https://raw.githubusercontent.com/rust-analyzer/metrics/master/metrics.json, > warning, it's a 5.5 MB unformatted JSON) > - we have an in-browser dashboard that retrieves the whole file and > displays them: https://rust-analyzer.github.io/metrics/ > - we do track build time and overall run time, but we're more interested > in correctness > - the display is a bit of a mess (partly due to trying to keep the setup > as simple as possible), but you can look for the "total time", "total > memory" and "build" to get an idea > - we store the runner CPU type and memory in that JSON; they're almost > all Intel, but they do upgrade from time to time > - we even have two AMD EPYC runs, note that boost is disabled in a > different way there (we don't try to disable it, though) > - we also try to measure the CPU instruction count (the perf counter), > but it doesn't work on GitHub and probably in most VMs > - the runners have been very reliable, but not really consistent in > performance > - a bigger problem for us was that somebody actually needs to look at the > dashboard to spot any regressions and investigate them (some are caused by > external changes) > - in 3-5 years we'll probably have to trim down the JSON or switch to a > different storage > > Laurentiu > > On Tue, Oct 10, 2023, at 21:08, Even Rouault via gdal-dev wrote: > > Hi, > > > > I'm experimenting with adding performance regression testing in our CI. > > Currently our CI has quite extensive functional coverage, but totally > > lacks performance testing. Given that we use pytest, I've spotted > > pytest-benchmark (https://pytest-benchmark.readthedocs.io/en/latest/) > as > > a likely good candidate framework. > > > > I've prototyped things in https://github.com/OSGeo/gdal/pull/8538 > > > > Basically, we now have a autotest/benchmark directory where performance > > tests can be written. > > > > Then in the CI, we checkout a reference commit, build it and run the > > performance test suite in --benchmark-save mode > > > > And then we run the performance test suite on the PR in > > --benchmark-compare mode with a --benchmark-compare-fail="mean:5%" > > criterion (which means that a test fails if its mean runtime is 5% > > slower than the reference one) > > > > From what I can see, pytest-benchmark behaves correctly if tests are > > removed or added (that is not failing, just skipping them during > > comparison). The only thing one should not do is modify an existing test > > w.r.t the reference branch. > > > > Does someone has practical experience of pytest-benchmark, in particular > > in CI setups? With virtualization, it is hard to guarantee that other > > things happening on the host running the VM might not interfer. Even > > locally on my own machine, I initially saw strong variations in timings, > > which can be reduced to acceptable deviation by disabling Intel > > Turboboost feature (echo 1 | sudo tee > > /sys/devices/system/cpu/intel_pstate/no_turbo) > > > > Even > > > > -- > > http://www.spatialys.com > > My software is free, but my time generally not. > > > > _______________________________________________ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > _______________________________________________ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev >
_______________________________________________ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
