On Tue, May 26, 2026 at 9:27 PM Branko Čibej <[email protected]> wrote:
> On 26. 5. 26 20:05, Branko Čibej wrote: > > I was looking at the test output of the CMake tests recently, and I > noticed that they don't run XML validation even when it's not explicitly > disabled. Looking at CMakeLists.txt, I found that each test program is run > directly instead of through run_tests.py. > > I don't think that's a very good idea. Sure, developers can run individual > tests like that, but run-tests.py adds some common semantics that the build > system shouldn't just skip. Like, for example, running XML validation on > command output. It also handles test parallelization. I understand that the > CMake project can run each test function from all the tests separately, but > it's possible to do that through run-tests.py. > > The alternative would probably be to install python-lxml and > python-rnc2rng on the Windows test runners. I already did that for Linux > runners (both autotools and cmake), because it is a bit more controlled > than creating a virtual environment on the fly. However, I don't (want to?) > know now to do that with vcpkg. > > > > Never mind. It was enough to run 'python -m pip' to install. > > Tests with cmake are basically unfinished... The problem is ctest (the utility that cmake advises to use for testing and what add_test command generates data for) is pretty much unusable. Sure, you can add tests and execute them, it even supports parallel execution and running specific tests, but there are limitations. There is no easy way to pass arguments to the test programs and no setup action. Now as I rethink that, it's probably a good idea to add a script similar to win-tests.py that runs tests when built with cmake. It would be also nice to add itself to either generator target or to ctest. There is an issue that if we want it to be possible to run the tests via this script directly (from the root of the source tree or something like that) it needs to be aware of where cmake was configured which with out-of-tree build is any directory on the filesystem. This script could accept this directory as an argument which also defaults to something like 'out' or 'out/build' and could be potentially cached for convenience. It might also be a good idea to factor out the part that starts an http server from win-tests.py into a separate python file to reuse it here and maybe even merge with davautocheck.sh. I also wonder why we don't use python's native unittest framework [1]? Now it's probably too late to suggest that but rather as a crazy idea, how about we switch to it? My reasoning would be so we don't reinvent the wheel. The wheel that worked well enough for a long time, but still. By using a standard framework like this some IDEs might automatically discover tests and display them in the UIs. I think it's nice to have every test as a first class entity if it makes sense that "adapters" can load, run, and list in whatever way they want. I'm pretty sure that swig tests use this framework instead of svntest. Thoughts? [1] https://docs.python.org/3/library/unittest.html -- Timofei Zhakov
