The one problem I have with this proposal is that this sounds like this
is proposing to do smoke test runs on every single pull request. That's
likely going to be too much overhead for our smoke test management.
For patch-specific testing, usually the reviewer handles doing the code
review of the patch and manually testing the patch to ensure that the
patch does what it intends to do and that the patch does not cause
severe regressions. However, there are times when there is value to get
QA involved on the specific patch (e.g. complex patch that has high
regression risk). In those cases, I think there's value to following a
process such as what's you've specified below to generate a try build
with the pull request and ask for QA support to do testing around the
patch to check for regressions, ensure the patch does it was says it
should do, etc.
I think there's a bug on file to add support for Gaia try builds to
implement this. Note that QA can manually generate a custom Gaia build
to follow the try build testing process right now, although having the
automated try build process would definitely make this process easier to
execute.
Sincerely,
Jason Smith
Firefox OS QA Engineer
Mozilla Corporation
https://quality.mozilla.com
On 6/11/2013 3:06 PM, Gareth Aye wrote:
Hi Everyone!
I wanted to propose an idea for how we can make the RIL builds more
relevant for and accessible to developers. Let me begin by saying that
I truly appreciate the work that QA does and that I do personally read
through all of the smoketest build results to see whether there are
any regressions my work has introduced. I think it's a great resource!
This is simply one idea for how we could make it an even better one.
I believe (and I think others would agree) that tests are most useful
when a patch is submitted for review. When I do a code review, I like
to see the linter and test results alongside the patch to get a full
perspective. Travis is actually *much* more useful to me than TBPL
because Travis tells me how patches affect things when it matters
most: when I'm reading code and deciding whether a patch is ready to
land. Imagine how many things we wouldn't have broken if reviewers
knew (when they accepted patches) whether or not they were breaking
the smoketests!
So, without further adieu, I propose that we do just that! Let's run
the smoketests on pull requests!
There's a catch though -- we have a very small QA team and there are
*lots* of pull requests. It's totally reasonable to question whether
this is possible. It's my opinion that with some clever automation we
could make the volume of QA work totally manageable. In broad strokes...
1. GitHub sends a notification to our service-x to tell us that a pull
request has been opened against mozilla-b2g/gaia.
2. service-x parses from the pull request information which bugzilla
issue the pull request is trying to fix.
3. service-x parses from the pull request which smoketests would be
affected by the patch.
4. service-x fetches the bug from bugzilla and parses out the
environment (ie gaia version and gecko version) that the patch is
intended to be applied to.
5. service-x cherry picks the patch onto the appropriate gaia branch
and builds b2g.
6. service-x sends a notification (maybe an email?) to qa with a link
to download the build, the smoketests that need to be run, and a link
to the pull request to comment on with the result of the smoketests.
I think if we built a service-x, it would make the amount of work that
qa needed to do in order to provide smoketest results to all of the
pull requests manageable. The manual work would be minimized to
flashing a build to a device, running only the affected smoketests,
and commenting on a GitHub issue.
Any thoughts :) ?
--
Best,
Gareth
_______________________________________________
Qa-b2g mailing list
[email protected]
https://mail.mozilla.org/listinfo/qa-b2g
_______________________________________________
dev-b2g mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-b2g
