Thanks for the reality check Jason && Tony :). I don't think I knew how resource-constrained QA was and I just figured this might help bridge what seems like a pretty big divide between eng and QA.
While we're on the topic, I would really appreciate if someone from QA wouldn't mind giving an intro to FFOS QA at some point. I'm clearly pretty clueless about what you guys are up to and I think I'm maybe not the only one? Some of us are very interested in adding more automated, integration-level tests to Gaia and it'd be nice if we could work together! On Tue, Jun 11, 2013 at 4:33 PM, Tony Chung <[email protected]> wrote: > To add to jason's points, It's absolutely too much overhead for my team > to smoketest each checkin. The only feasible way is having automated tests > that run per checkin. This is worth discussing with the A-team / Gaia UI > team if that is feasible, and in what time frame. > > our current QA team does write automated tests, but it currently supports > 1.01 smoketests and we havent gotten around to testing new features. We're > still one build behind and can't catch up current release fast enough for > this. We would request the help of gaia dev in this effort. (device > level testing is better, but maybe running on desktop build is good enough) > > The other condition you need is parallelizing gecko changes as gaia > changes land. I know some checkins are independent of gecko, but some > aren't and require parallel landings. To test end to end infra per > checkin, you'd need to sync up landings with gecko. At least within 24 > hours i suppose. This means if we sync master, we'll have to sync > mozilla-central as well, to truly get integration testing going. > > Tony > > On Jun 11, 2013, at 3:22 PM, Jason Smith <[email protected]> wrote: > > The one problem I have with this proposal is that this sounds like this > is proposing to do smoke test runs on every single pull request. That's > likely going to be too much overhead for our smoke test management. > > > > For patch-specific testing, usually the reviewer handles doing the code > review of the patch and manually testing the patch to ensure that the patch > does what it intends to do and that the patch does not cause severe > regressions. However, there are times when there is value to get QA > involved on the specific patch (e.g. complex patch that has high regression > risk). In those cases, I think there's value to following a process such as > what's you've specified below to generate a try build with the pull request > and ask for QA support to do testing around the patch to check for > regressions, ensure the patch does it was says it should do, etc. > > I think there's a bug on file to add support for Gaia try builds to > implement this. Note that QA can manually generate a custom Gaia build to > follow the try build testing process right now, although having the > automated try build process would definitely make this process easier to > execute. > > Sincerely, > Jason Smith > > Firefox OS QA Engineer > Mozilla Corporationhttps://quality.mozilla.com > > On 6/11/2013 3:06 PM, Gareth Aye wrote: > > Hi Everyone! > > I wanted to propose an idea for how we can make the RIL builds more > relevant for and accessible to developers. Let me begin by saying that I > truly appreciate the work that QA does and that I do personally read > through all of the smoketest build results to see whether there are any > regressions my work has introduced. I think it's a great resource! This is > simply one idea for how we could make it an even better one. > > I believe (and I think others would agree) that tests are most useful > when a patch is submitted for review. When I do a code review, I like to > see the linter and test results alongside the patch to get a full > perspective. Travis is actually *much* more useful to me than TBPL because > Travis tells me how patches affect things when it matters most: when I'm > reading code and deciding whether a patch is ready to land. Imagine how > many things we wouldn't have broken if reviewers knew (when they accepted > patches) whether or not they were breaking the smoketests! > > So, without further adieu, I propose that we do just that! Let's run the > smoketests on pull requests! > > There's a catch though -- we have a very small QA team and there are > *lots* of pull requests. It's totally reasonable to question whether this > is possible. It's my opinion that with some clever automation we could make > the volume of QA work totally manageable. In broad strokes... > > 1. GitHub sends a notification to our service-x to tell us that a pull > request has been opened against mozilla-b2g/gaia. > 2. service-x parses from the pull request information which bugzilla > issue the pull request is trying to fix. > 3. service-x parses from the pull request which smoketests would be > affected by the patch. > 4. service-x fetches the bug from bugzilla and parses out the > environment (ie gaia version and gecko version) that the patch is intended > to be applied to. > 5. service-x cherry picks the patch onto the appropriate gaia branch and > builds b2g. > 6. service-x sends a notification (maybe an email?) to qa with a link to > download the build, the smoketests that need to be run, and a link to the > pull request to comment on with the result of the smoketests. > > I think if we built a service-x, it would make the amount of work that > qa needed to do in order to provide smoketest results to all of the pull > requests manageable. The manual work would be minimized to flashing a build > to a device, running only the affected smoketests, and commenting on a > GitHub issue. > > Any thoughts :) ? > > -- > Best, > Gareth > > > > _______________________________________________ > Qa-b2g mailing [email protected]https://mail.mozilla.org/listinfo/qa-b2g > > > _______________________________________________ > B2g-release-drivers mailing list > [email protected] > https://mail.mozilla.org/listinfo/b2g-release-drivers > > > -- Best, Gareth _______________________________________________ dev-b2g mailing list [email protected] https://lists.mozilla.org/listinfo/dev-b2g
