On 6/2/13 10:18 AM, "Erik de Bruin" <e...@ixsoftware.nl> wrote:
>Does that imply that once you cut the new baselines ALL tests will >pass? That sounds cool! I've never seen a fully successful run of >Mustella ;-) I've only seen about 4. It looks like you are still getting some timeouts. But for me on the Mac and Win on Friday, running -all followed by -failures ended up with 0 failures, but I realized after that I hadn't pulled changes since earlier that week. So, the metric for success may require that script that checks for failures.txt and runs with the -failures options. > >Once you get all tests to pass I'll restart the twice daily schedule >and we will finally have a max of 12 hours between commit and a signal >something broke the SDK. Let the contributions start streaming in! > >Also, this should help us a long way towards 4.10, shouldn't it? Yup, and I'll get the patch tester up and running early next week. We had both at Adobe. The patch server took a .patch file, figured out the minimum set of tests to run, ran them, and sent you back results. That has the advantage of 1) You find out in private that you broke something 2) You should find out sooner than 12 hours, although a change to TextBase.as took 4 hours. 3) If you submit a patch and it breaks something, you don't have to scramble to revert your change, since all future changes are going to break until the bad change is found an reverted. The patch server reverts the patch after the run so there isn't cross-pollinization. 4) You know for sure it was you. At 12 hour intervals, several changes from several folks get tested at once. 5) Non-committers can test their patches before submitting. But for sure we need a commits-tester, which is what I call what you've set up since the patch-tester is optional. We can't force folks to use it and the only thing that really matters is whether the code in the repo passes or not. But the interesting thing for me once we get this Jenkins thing running is: if several committers also set up this same Jenkins job, could we have them each run a subset of the mustella tests and report much sooner? I think it was you who suggested we could create a massively parallel mustella system. -Alex
