Hey People! This was a bad week for gaia. I wrote about it though! Perhaps we can make future weeks better! I wanted to write in markdown, so if your email client is unhappy, you can also read here on GitHub<https://gist.github.com/gaye/9127473#file-gaia-tree-closure-post-mortem-md> . Gaia Tree Closure Post-MortemWhat Happened
1. On Tuesday, we noticed the js ui tests were erroring starting at this build <https://travis-ci.org/mozilla-b2g/gaia/builds/18901708>. 2. We closed the tree to investigate. 3. An npm outage started almost immediately after the tree was closed. Our travis builds were still having issues fetching our dependencies from npm last night when 4. Kevin Grandon awesomely moved our dependencies onto GitHub<https://github.com/mozilla-b2g/gaia-node-modules> . 5. We narrowed down the issue to a b2g-desktop crash triggered (intermittently) by sending emails from the email app (we use a fake imap server in our integration tests). 6. After failing to identify the patch that introduced the regression or the the reason b2g-desktop was crashing, we felt it was time to reopen the tree. What We're Already Doing Differently When we built our ui testing framework, we put all of our dependencies in npm and made gaia dependent on the health of the npm registry. The registry has been plagued with outages that have hurt us in the past year. We have reached the point where we are kidding ourselves if we continue to expect for them to have the high availability we need for our testing infrastructure. Last night, :kgrandon moved our dependencies into a repo in GitHub. We are now, as far as external services go, only relying on GitHub and Travis for testing gaia. How Can We Do Things Differently in the Future Closing the tree sucks. This week (AFAICT) our project came to a screeching halt because of it. We absolutely should work to lessen the frequency and severity of these incidents. I would like to seed the conversation about what needs to be done with some ideas for what can be done. Some of these may seem vacuously smarter than others, but (in the spirit of keeping my personal biases from directing the conversation) I want to put everything on the table. Idea #1: The Build Needn't Always Be 100% Green For a long time at the end of 2013, the build was mostly broken. We made a habit of reading the test results as { green: "Build works!", grey: "Build needs to be run again!", red: "Build might not work!" } Our tests were less informative, but the tree stayed open. Nowadays, we judge greys and reds more harshly which keeps our build much greener but also pushes us to more extreme measures when tests fail inexplicably and/or intermittently. Idea #2: Instead of Closing the Tree, Disable Tests and File Bugs Why do we close the tree when we find regressions? Because the broken tests can no longer keep the corresponding features' functionality from regressing even further. Closing the tree is like wearing an astronaut's suit when you have an autoimmune disease. You simply *cannot* risk being exposed to more bugs. But maybe we can risk being exposed to more bugs. One idea is that, whenever a regression pops up, we can simply disable the broken test and file a high priority bug to diagnose the regression, revert an offending patch or submit a fix, and re-enable the test. Idea #3: Always Throttle the Tests to Proactively Discover Intermittent Failures Why do we have regressions anyway? Who lands broken code in our tree? Can't we scold them and get on with our lives? Well, it's not that simple for the following two reasons. 1. Some tests only fail some of the time. 2. Projects which we're downstrseam from don't pay attention when they break us. I'll address the first issue here and the second one in the next section. One idea that I've been championing on the mailing lists (and :evanxd has recently introduced a patch to automate) is that we throttle tests on checkins. Suppose Bob wrote the following contrived test using our most favorite test harness <http://visionmedia.github.io/mocha/>: test('should work', function() { assert.ok(Math.random() > 0.5); }); Then suppose further that Bob submitted a patch with his test, saw his pull request pass on CI, and merged his code. Then, all of a sudden, his test started burning when Alice checked in a completely unrelated patch 10 minutes later. *Oh noes!* If we had setup our CI to run Bob's test enough times to tell with statistcal significance that it was passing, the whole debacle could have been avoided! I've encouraged people to do this on an ad hoc basis, but maybe if (Travis and/or :lightsofapollo) ever solve our testing capacity problems, we should make throttling tests a normal practice. On the one hand, machine time is expensive. On the other hand, so is developer time. Idea #4: Make our JS UI Tests Visible on TBPL My next idea, which I alluded to earlier, is that we prevent gecko patches which break gaia from landing. We have setup our js marionette tests to run downstream from b2g-desktop builds on TBPL to do exactly this, however our test are currently hidden on TBPL. In my opinion, things are relatively stable, but we have sheriffs who may strongly disagree<https://bugzilla.mozilla.org/show_bug.cgi?id=960072#c8>? I would not be surprised if we are stable enough to be a visible test suite at this point, and I strongly recommend we look into unhiding these tests. Idea #5: Invest in Infrastructure Which Helps Fix Regressions No matter how good our process and automation gets, we will have regressions. We have a *very* complex project with lots of dependencies and sometimes things just fall through the cracks. There are several tools that might have helped us recover faster this week which either haven't been built or haven't yet been used heavily in gaia. Some examples are: - Tools for gaia/gecko bisection (meaning to look into :jhford's project) - Crash reporting for js ui tests - Tools uploading, sorting, and grouping screenshots taken via marionette -- Best, Gareth _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
