On Sat, Nov 30, 2013 at 2:19 PM, Daniel Shahaf <d...@daniel.shahaf.name>wrote:
> Stefan Fuhrmann wrote on Sat, Nov 30, 2013 at 08:48:30 +0100: > > Again, you are missing the point. You are saying that there > > are lots of possible configurations and various combinations. > > I'm saying that we *are* actually able to test many variants. > > One simply needs to update their overall test script. > > There are classes of problems that our test suite simply isn't designed > to catch. > > For example: > > - We won't catch 'premature kill -9' bugs (r1494298) > - We won't catch concurrency bugs (r1302613) > It is true that our test suite does not cover these scenarios. This has always been the case and so far did not prevent releases. My private testing with larger repositories (a.o. & friends) had me identify a concurrency / consistency issue with 'svnadmin pack' during a 'svnadmin load' In the future, however, we can of course add tests that either kill some test application or trigger some yet-to-be- designed kill / suicide callbacks in the code. My recent change to make the C tests execute in parallel is to some extend intended to simply put "unusual" stress on our code. And that identified a few points where our code or APR did not behave as expected by me at least. 1.8 added the ability to generate code coverage reports for our tests (not sure that currently works). So, we should also be able to identify critical bits that don't get any coverage. > So, no, running "make check" under more configurations (despite still > being an improvement) won't make me more confident that FSFS7 is in > release-shape. It simply doesn't test enough code paths. > Well, Ivan's argument was along the lines of "many formats" and their combination over the lifetime of a given repository. That's a fair point and this one *can* be addressed be running our test suite through many (all?) of these combinations. BTW, the log-addressing branch is not the first incarnation of that code. Developed in the format7 branch, it underwent review (by me) and improvement while it got transferred to the log-addressing branch. There it received stress testing and further fixes. Finally, Philip gave it an in-depth review with a specific focus on potential concurrency issues. A few minor things showed up and got fixed. Now, we are at a point were internal review is certainly welcome and will probably find more issues but the real test is using it in the real world, e.g. as part of an alpha release. -- Stefan^2.
