On Thu, Mar 10, 2011 at 11:36 AM, James Westby <james.wes...@canonical.com> wrote: > On Thu, 10 Mar 2011 11:09:21 +1300, Michael Hope <michael.h...@linaro.org> > wrote: >> > * How frequently do problems get through review that are found by the >> > test suite? >> >> Ideally the test suite should have been run before the review and have >> shown no regressions. That's how upstream does it. GCC is >> complicated so a review will miss subtle things. > > Sure, but reviews shouldn't be trying to catch every single problem that > may exist. > > If you don't have an idea of how much benefit test-before-merge has over > just review-before-merge then it's going to be hard to estimate the risk > of changing that scheme.
We currently have review+build+test before merge. If we checkin directly to trunk then it would be review+merge, then build+test automatically afterwards. >> > * How frequently do problems get past the testsuite? >> >> Being pessamistic, say every second patch causes a failure found in >> the field not found in the testsuite. > > That indicates that getting the code out there for (non-production) use > would actually be worthwhile, rather than trying too hard to ensure > everything goes through the testsuite. (Unless every patch has 10 > problems that are not found in review but found by the testsuite.) Yip. It's quite stable and bugs are quite rare. A huge testsuite like Ubuntu is a good way of exercising the compiler. >> > * Would you be confident in your ability to have quality releases if >> > there wasn't a testsuite run of each change before it lands in >> > trunk? >> >> Not a quality release, no, but I have good confidence in the patches >> in general. It's unlikely that a commit would break trunk. > > What sense are you using "break" here? On second reading, too loosely. I meant 'fail to build', but most developers will want to run the testsuite. If the checkin before them caused a testsuite regression then they may get confused. A timely automatic build would help with that: perhaps notify the previous author of the regression, and give the developer somewhere to easily check the state of their baseline. > It sounds like prompt review->integration branch->beta testers > | > `->test suite->trunk > > would be best if you don't think that you can have a quality release if > you merge-before-test. Yip. The integration branch however may be Ubuntu itself. Matthias is fairly happy cherry-picking things so if these checkins/feature branches are at least reviewed and preferably build+tested then he can be the path to beta testers. This may change when Linaro gets set up to build main. > If you feel that you are equipped to recover from problems found by > running the test suite against trunk before doing a release then it may > be better to avoid the overhead and merge to trunk after prompt review, > and deal with test failures when they happen. Agreed. The build+testsuite can happen automatically and probably within two days. -- Michae _______________________________________________ linaro-dev mailing list linaro-dev@lists.linaro.org http://lists.linaro.org/mailman/listinfo/linaro-dev
