On Sat, Nov 30, 2013 at 2:23 PM, Daniel Shahaf <d...@daniel.shahaf.name>wrote:
> Daniel Shahaf wrote on Sat, Nov 30, 2013 at 15:19:56 +0200: > > Stefan Fuhrmann wrote on Sat, Nov 30, 2013 at 08:48:30 +0100: > > > Again, you are missing the point. You are saying that there > > > are lots of possible configurations and various combinations. > > > I'm saying that we *are* actually able to test many variants. > > > One simply needs to update their overall test script. > > > > There are classes of problems that our test suite simply isn't designed > > to catch. > > > > For example: > > > > - We won't catch 'premature kill -9' bugs (r1494298) > > - We won't catch concurrency bugs (r1302613) > > As another example, we have architectue-dependent code, for example > the definition of SVN_UNALIGNED_ACCESS_IS_OK. In those cases, do we > test the "unlikely" branch (eg: SVN_UNALIGNED_ACCESS_IS_OK=0) too? > Well, that is exactly the kind of things that a test script should cover. Running just another combination of config options add less than 2:30 total (or 2 minutes with ra_local). So, being able to quickly run a great number of variants is clearly useful here. > My build script defines -DSVN_UNALIGNED_ACCESS_IS_OK=0 for precisely > this reason. > I think we should make a list of combinations that should be part of release testing. It is certainly a waste to do something like [local, serf, svn] x [bdb, fsfs, fsx] x [1.0 .. 1.9] x [P(config options)] Instead, maybe along the lines of: [local, serf, svn] x [bdb, fsfs, fsx] x [1.9] x [config defaults] [local, serf or svn] x [fsfs] x [1.9] x [Selected config option sets] [local, serf or svn] x [fsfs] x [1.1 .. 1.9] x [config defaults] [local, serf or svn] x [bdb] x [1.0] x [config defaults] For the ra layer bit, I feel that having "all in one" and "separate c/s" may trigger different failure scenarios. BDB is deprecated and FSX is experimental, so let's focus on FSFS for the non-standard setups. 1.9 should cover the most code making it a good test bed for config option variation. At least the standard config should work with all format versions. Makes sense? What combinations of config options should we test? -- Stefan^2.
