Hi, On Fri, Mar 21, 2014 at 8:32 AM, Nicholas Skaggs < nicholas.ska...@canonical.com> wrote:
> > As an example, the calendar blockage on the latest image has a few tests > that started failing after 5.2 landed. You've seen the landing emails with > this snippet; > > * flaky test failure on calendar-app (Alan/Jean-Baptiste): > https://bugs.launchpad.net/ubuntu-calendar-app/+bug/1293489 > > To fix you pull calendar trunk and investigate the tests. But wait, you > find out there's a regression caused by qtorganizer that causes calendar > trunk to be broken. Now before you can fix the image blocker, you need to > fix the calendar app trunk. In the meantime nothing can land in calendar > trunk, and no fix can land in the image. Repeat for each regression you > find while stabilizing trunk and you can understand the source of the > delays. I agree - to summarise balloon's point: One cannot assume that an acceptance test failure in application X represents a regression in application X. Often it's actually in low-level-component Y. This is a symptom of having large test coverage gaps for lower level components. It's one of the reasons why, at the recent UDS, we pushed for decoupling the trunk branch from the release branch. At least then, under that proposal, development teams can continue to develop their projects, and landing managers are still responsible for releasing components without regression to the image. It doesn't solve the problem of "I can't release my app until the existing failing tests are fixed", but it *does* solve the problem of "My development team cannot (easily) write any more code while there are failing tests in the image". The latter problem sounds like a good thing, if you make the assumption that failing tests in app X represents a failure in app X. As we've discovered many times in the last cycle, that's actually reasonably rare (at least, it's rare with our test coverage being the way it is. I'm sure this is improving already). Cheers, -- Thomi Richards thomi.richa...@canonical.com
-- Mailing list: https://launchpad.net/~ubuntu-phone Post to : ubuntu-phone@lists.launchpad.net Unsubscribe : https://launchpad.net/~ubuntu-phone More help : https://help.launchpad.net/ListHelp
