On Fri, Aug 10, 2012 at 11:40 PM, Anthony Hughes <ahug...@mozilla.com> wrote:
I'm commenting only from the point of view of developing Web-exposed features into Gecko. I don't have sufficient experience to comment on QA practices as they relate to Firefox UI development, etc. > Does verifying as many fixes as we do really raise the quality bar for > Firefox? I think verifying that the steps to reproduce that the bug reporter stated no longer reproduce the bug is not a good use of QA time. While it is possible for the developer to make a mistake, most of the time bugs don't get marked FIXED without actually landing something and developers test their patches with the steps to reproduce that were reported. Therefore, I would expect it to be very unlikely for verification of the original steps to reproduce to result in a quality-improving action. > Could the time we spend be better used elsewhere? I think we have a lot to learn from Opera here. When we develop Web-exposed features in Gecko, typically the test cases that get landed together with the patch are written by the same developer who wrote the patch and QA isn't involved at all. This means that the testing of the code is limited by the imagination of the person who wrote the code being tested. If the person who wrote the code didn't think of handling an edge case in the code, (s)he probably didn't think of the edge case enough to write a test for it. We (mostly) send Gecko developers to participate in Web standardization. Opera (mostly) sends QA people. This results in Opera QA having a very deep knowledge and understanding of Web standards. (I'm not suggesting that we should stop sending Gecko developers to participate. I think increasing QA attention on spec development could be beneficial to us.) It seems (I'm making inferences from outside Opera; I don't really know what's going on inside Opera) that when a new Web platform feature is being added to Presto, Opera assigns the QA person who has paid close attention to the standardization of the feature to write test cases for the feature. This way, the cases that get tested aren't limited by the imagination of the person who writes the implementation. So instead of verifying that patches no longer make bugs reproduce with it steps to reproduce provided by the bug reporter, I think QA time would be better used by getting to know a spec, writing Mochitest-independent cross-browser test cases suitable for contribution to an official test suite for the spec, running not only Firefox but also other browsers against the tests and filing spec bugs or Firefox bugs as appropriate (with the test case imported from the official test suite to our test suite). (It's important to sanity-check the spec by seeing what other browsers do. It would be harmful for Firefox to change to match the spec if the spec is fictional and Firefox already matches the other browsers.) -- Henri Sivonen hsivo...@iki.fi http://hsivonen.iki.fi/ _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
