vsavchenko added a comment. In D86295#2233157 <https://reviews.llvm.org/D86295#2233157>, @steakhal wrote:
> In D86295#2233076 <https://reviews.llvm.org/D86295#2233076>, @vsavchenko > wrote: > >> In D86295#2231760 <https://reviews.llvm.org/D86295#2231760>, @NoQ wrote: >> >>> I believe @vsavchenko's docker thingy already knows how to do that. >> >> Yep, it sure does! Additionally, there is a `benchmark` subcommand that can >> chart memory consumption for measured projects. > > Could you point me to some docs to help getting started? I haven't used this > docker image. I procrastinated the part about docs 😅 I will add those in the nearest future. Here is a short summary how to do regression testing (check that all warnings are the same): 1. `cd clang/utils/analyzer` 2. Build test system's docker image (I assume that `docker` is installed): `./SATest.py docker --build-image` 3. Checkout commit that you want to compare against: `git checkout origin/master` or `git checkout HEAD^1` 4. Build LLVM in docker for Ubuntu (if you are on Linux you might not it): `./SATest.py docker --clang-dir {PATH_WHERE_IT_WILL_BE_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- --build-llvm-only`. I usually put `--build-dir` next to my native build directories, ie `${LLVM_DIR}/build/Docker` 5. Gather reference results: `./SATest.py docker --clang-dir {PATH_WHERE_LLVM_IS_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- build -j1 -r` 6. Checkout your commit that you want to test: Checkout commit that you want to compare against: `git checkout master` or similar 7. Build LLVM with your patch: `./SATest.py docker --clang-dir {PATH_WHERE_IT_WILL_BE_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- --build-llvm-only`. If you choose the same `--build-dir` it will get compiled incrementally and save you a lot of time. TIP: you can choose another directory for `--clang-dir` so you don't need to rebuild previous version if you have to re-run it. 8. Gather new results and compare: `./SATest.py docker --clang-dir {PATH_WHERE_LLVM_IS_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- build -j1 --strictness 2`. NOTE: `SATest.py docker ... -- build` command accepts a comma-separated list of projects for the analysis, so you can choose which projects to test. Benchmarking is very similar in it's nature, run original, run modified, compare: 5. `./SATest.py docker --clang-dir {PATH_WHERE_LLVM_IS_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- benchmark -o old.csv` 6. ... 7. ... 8. `./SATest.py docker --clang-dir {PATH_WHERE_LLVM_IS_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- benchmark -o new.csv` 9. `./SATest.py docker --clang-dir {PATH_WHERE_LLVM_IS_INSTALLED} --build-dir {PATH_FOR_BUILD_DIR} -- benchmark compare --old old.csv --new new.csv -o comparison.png` Benchmarks can take a while (they run analysis on the same projects multiple times), so it is recommended to use it for smaller projects. I will merge D83942 <https://reviews.llvm.org/D83942> and after it you can change commands this way: `./SATest.py docker ... benchmark --max-size small -o blabla.csv` If you want to see `--help` for a subcommand, drop `docker` and all the options and simply run `./SATest.py build --help` or `./SATest.py benchmark --help`. Repository: rG LLVM Github Monorepo CHANGES SINCE LAST ACTION https://reviews.llvm.org/D86295/new/ https://reviews.llvm.org/D86295 _______________________________________________ cfe-commits mailing list cfe-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
