Recently, in another thread ("PMCs: any Hackathon requests? (deadline 11
October)")...On Thu, Oct 10, 2019 at 4:54 PM Daniel Shahaf <d...@daniel.shahaf.name> wrote: > Something in the test harness. For example, make it easier to run «make > check» > with client version ≠ server version, to actually test on-the-wire > compatibility. Testing with different client and server versions has been mentioned here several times recently. I've been giving this some thought. I think this is important given how svn is used in the real world. How would we do this? I assume it would be something along these lines: A test "driver" program would contain a list of versions to be tested. That program would download, configure, make, make check, and make install each listed version to a different prefix. Then, it would iterate through all permutations of client and server versions (except equal client/server versions, since those are already tested by "make check") and run the tests. Actually, all permutations sounds like overkill and would take an unreasonable length of time. We would probably test the latest client against all listed server versions, and the latest server against all listed client versions. Is this a reasonable initial concept? If so, answers / solutions are needed for the following: (1) Which versions are we interested in cross-testing in this manner? Do we want to limit ourselves to only cross-testing currently supported versions? Do we want to test unsupported versions that are likely to be in reasonably widespread use today, including 1.8.x and 1.9.x? [1] Do we want to go as far back as some antique version like 1.5 (e.g. test a 1.13 client against a 1.5 server; test a 1.5 client against a 1.13 server)? Do we want to go for ultimate flexibility and allow testing any two trunk and/or branch revisions against each other (which is different than, say, testing released or rc code from tarballs). Do we want this to be configurable, i.e. the tester could choose a "shallow" or "deep" test? (2) How do we handle differences between versions? For example, newer versions probably contain more features and their associated tests, and more tests in general than older versions. Is the test driver program supposed to contain knowledge of these differences and prevent some of the tests from running under certain combinations of client and server versions? (3) How do we handle dependencies? For example IIRC until some recent version we couldn't build against APR 1.7.0. Now we can. Do we try to find a least common denominator version of each dependency and build all versions with that? Or is it better to build each version with the dependency versions as listed in get-deps.sh? Am I on the right track? Nathan Notes: [1] I'm basing that on what's in a certain popular OS's package manager and recent messages to users@.
