On Fri, Mar 20, 2015 at 10:30 AM, Johan Corveleyn <jcor...@gmail.com> wrote:
> On Fri, Mar 20, 2015 at 9:42 AM, Johan Corveleyn <jcor...@gmail.com> > wrote: > ... > > Unfortunately, I can't verify the rev > > file, since I don't have it anymore, it has been overwritten by trying > > to reproduce it (grrrr, should remember to backup leftover > > repositories and working copies after a failed test run, before trying > > to reproduce it). Whatever I try now, I can't reproduce it anymore > > :-(. > > I'm wondering if something can be improved in our test suite to help > diagnosis of hard-to-reproduce test failures. When this happens, you > typically wish you could analyse as much data as possible (i.e. the > potentially corrupt repository, working copy, dump file, ... that was > used in the test). > > Currently, I can think of three causes for losing this information: > > 1) You run a series of test runs in sequence from a script > (ra_local, ra_svn, ra_serf), all using the same target directory for > running the tests (R:\test in my case, where R: is a ram drive). If > something fails in ra_svn, but succeeds in ra_serf, your broken test > data is overwritten. > > 2) You don't know in advance that the failure will turn out to be > non-reproducible. You can't believe your eyes, try to run it again to > be sure, and lo and behold, the test succeeds (and the broken test > data is overwritten), and succeeds ever since. > > 3) Your test data is on a RAM drive, and you reboot or something. Or > you copy the data to a fixed disk afterwards, but lose a bit of > information because last-modified timestamps of the copied files are > reset by copying them between disks. > > > For 1, maybe the outer script could detect that ra_svn had a failure, > and stop there (does win-tests.py emit an exit code != 0 if there is a > test failure? That would make it easy. Otherwise the outer script > would have to parse the test summary output)? > > Another option is to let every separate test run (ra_local, ra_svn, > ra_serf) use a distinct target test directory. But if you're running > them on a RAM disk, theoretically you might need three times the > storage (hm, maybe not, because --cleanup ensures that successful test > data is cleaned up, so as long as you don't run the three ways in > parallel, it should be fine). I guess I will do that already, and > adjust my script accordingly. > > > Addressing 2 seems harder. Can the second test execution, on > encountering stale test data, put that data aside instead of > overwriting it? Or maybe every test execution can use a unique naming > pattern (with a timestamp or a pid) so it doesn't overwrite previous > data? Both approaches would leak data from failed test runs of course, > but that's more or less the point. OTOH, you don't know that stale > test data is from a previous failed run, or from a successful run that > did not use --cleanup. > > > And 3, well, I already reboot as little as possible, so this is more > just a point of attention. > > > Maybe addressing all three at the same time could also be: after a > failed test run, automatically zip the test data and copy it to a safe > location (e.g. the user's home dir). > > Thoughts? > I haven't thought too deeply about it but I think we should be able to extend the current repo / w/c cleanup infrastructure to copy the data away upon test failure. -- Stefan^2.
