add U-Boot tests on real hardware with tbot, which is started from gitlab CI (or other CI implementations like jenkins, github,...)
Currently only one board in my lab@hungary is implemented, if series gets accepted more will follow. May others are interested too, and prepare also such setups ... or simply send me hardware, I can add to my setup. Current board is the imx8qxp based cxg3 board. To build and get a working image you need some binary blobs, which are copied from the lab host to the "CI host". After successfull build the resulting images are copied to the lab host, and new flash.bin is booted with uuu tool in USB SDP bootmode, and than installed with fastboot to the emmc. Than power is switched off, bootmode set to emmc and it gets checked, that U-Boot boots from emmc now. Than test/py gets configured and started on the CI host which runs the tests on the board in tha lab. Hopefully the approach is generic enough to get other hardware easy integrated, but if there are problems, just contact me, if stuff is missing. You can find a working example @gitlab in my i2c custodian branch here: https://source.denx.de/u-boot/custodians/u-boot-i2c/-/jobs/1081203 (If you have no access rights, look below for a test started @github) I dropped from above repo/branch the following commits in this series: - tbottest: add github example as it is not interesting for U-Boot mainline (or?) Just for interest, with that the same test is started from repo @github, example: https://github.com/hsdenx/u-boot-test/actions/runs/14212012371/job/39821029362 to see the full logs open there (Run U-Boot test) - .gitlab-ci.yml: only start tbot tests - remove .azure-pipelines.yml as this changes are only for speeding up my local testing. The patches in this series: - test/py/requirements.txt: update coverage - tools/buildman/requirements.txt: update coverage - test/py/requirements.txt: increase filelock version - test/py/requirements.txt: remove unittest2 and testtools - test: increase timeouts are fixing small common issues I found with test/py setup ... which could be go seperate into mainline, but in the first shot, I want to have all patches in one series. - CI: add U-Boot testing on real HW with tbot adds all the config needed for testing the imx8qxp based cxg3 board with tbot in tbottest subdirectory. We should discuss where to put this code! May into u-boot:/test/tbottest ? This setup not only runs on gitlab, it also runs from github, jenkins ... or from a local shell without any changes in code. So you can use it for your normal development process (what was the goal from tbot at the beginning, to be more precise, it was may laziness to type a lot of commands on different machines again and again). So if you use this setup for your daily work you get the CI integration for free! - .gitlab-ci.yml: add tbot-lab finally integrates the tbot test setup into .gitlab-ci.yml This works currently for my I2C custodian repo, with some CI secrets stored... we have to add this secrets to u-boot repo to get it up and running. If this is not possible may I should setup a gitlab runner? (Which I would like to prevent... but possible of course, and this would prevent the timing issues I fixed when tbot is started somewhere in the world) Bonus patch with an idea for custodians I want to implement, if code gets accepted: - scripts: add helper for getting infos from your patchwork ToDo list is a preparation for activating my 8 year old setup, which downloaded patches from my Patchwork ToDo list, checked them with checkpatch.pl script and applied them with git am, fully automated from within the CI ! With that script it is very easy to reactivate this now again, if this is interesting for others. Currently I have not a clean patchwork ToDo list, which is the reson for adding this not yet. My plan is also to reactivate an automated git bisect session, in case there are problems with that patches, so test shows the problematic patch at the end... which I think would be a great feature and benefit to have in mainline CI. For reference an old video on youtube how this worked 8 years ago: https://www.youtube.com/watch?v=PhaYfqOrQOg It would be nice, we can make such a step configurable through a switch we can enable/disable before starting a gitlab CI run. May time is now ready for it, that people have interests in such a setup? Heiko Schocher (8): test/py/requirements.txt: update coverage tools/buildman/requirements.txt: update coverage test/py/requirements.txt: increase filelock version test/py/requirements.txt: remove unittest2 and testtools test: increase timeouts [RFC] CI: add U-Boot testing on real HW with tbot .gitlab-ci.yml: add tbot-lab scripts: add helper for getting infos from your patchwork ToDo list .gitlab-ci.yml | 74 ++++ MAINTAINERS | 6 + scripts/getpatchlist.py | 126 +++++++ tbottesting/README.md | 323 ++++++++++++++++++ tbottesting/boardtestconfig.py | 18 + tbottesting/boardtests.py | 50 +++ tbottesting/preparetbot.sh | 5 + tbottesting/tbotboardtests.py | 127 +++++++ tbottesting/tbotconfig-hs/boardspecific.py | 111 ++++++ tbottesting/tbotconfig-hs/hs/README.cxg3 | 28 ++ tbottesting/tbotconfig-hs/hs/args/argsbase | 11 + tbottesting/tbotconfig-hs/hs/args/argshs | 1 + .../tbotconfig-hs/hs/args/argshs-github-ci | 7 + .../tbotconfig-hs/hs/args/argshs-noeth | 3 + .../tbotconfig-hs/hs/args/argshs-noeth-ssh | 5 + tbottesting/tbotconfig-hs/hs/cxg3.ini | 106 ++++++ tbottesting/tbotconfig-hs/hs/tbot.ini | 117 +++++++ tbottesting/tbotconfig-hs/interactive.py | 59 ++++ tbottesting/tbotconfig-hs/labcallbacks.py | 76 +++++ tbottesting/tbotconfig-hs/tc_cxg3.py | 82 +++++ test/py/requirements.txt | 6 +- test/py/u_boot_console_base.py | 4 +- tools/buildman/requirements.txt | 2 +- 23 files changed, 1340 insertions(+), 7 deletions(-) create mode 100755 scripts/getpatchlist.py create mode 100644 tbottesting/README.md create mode 100644 tbottesting/boardtestconfig.py create mode 100644 tbottesting/boardtests.py create mode 100755 tbottesting/preparetbot.sh create mode 100644 tbottesting/tbotboardtests.py create mode 100644 tbottesting/tbotconfig-hs/boardspecific.py create mode 100644 tbottesting/tbotconfig-hs/hs/README.cxg3 create mode 100644 tbottesting/tbotconfig-hs/hs/args/argsbase create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-github-ci create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-noeth create mode 100644 tbottesting/tbotconfig-hs/hs/args/argshs-noeth-ssh create mode 100644 tbottesting/tbotconfig-hs/hs/cxg3.ini create mode 100644 tbottesting/tbotconfig-hs/hs/tbot.ini create mode 100644 tbottesting/tbotconfig-hs/interactive.py create mode 100644 tbottesting/tbotconfig-hs/labcallbacks.py create mode 100644 tbottesting/tbotconfig-hs/tc_cxg3.py -- 2.20.1
