On 12-08-10 5:04 PM, Jason Smith wrote:
Hi Everyone,
Let's try posting this again. Disregard the comments I put on the other
thread.
I think this is a good time to re-think our process for testing for
something that is fixed or not fixed. I think a better approach that
maybe we need to consider is something similar to what the security
reviews do. They flag certain special areas for needing attention from a
security perspective to do "deep dives" to find out the risks of certain
area and flag them as such. In the context of QA, we could easily do
something like that by reusing that process with the qawanted keyword.
Some suggestions to build off of this:
* Continue doing the process when a try build is created and needing
QA testing to make use of the qawanted keyword to call out for
explicit QA help.
* Add a new process where more testing is needed after a bug has
landed for some purpose. We could reuse the qawanted keyword to call
attention to the bug and we can take it from there. For our QA
process, we might want to file bugs in our component (Mozilla QA) to
track any specialized testing requests that are needed
This sounds good.
* Throw out the general concept of blind verifications. Maybe we
should even take out the verified portion of the bug workflow to
ensure this, but that may need further discussion to get agreement
on that.
This^^^, a thousand times! :-)
Note that not everything to test is necessarily testing that "X bug
works as expected." There's other cases that may warrant calling out
attention to certain bugs for explicit testing after the bug lands. Here
are two examples:
* Unprefixing of a DOM or CSS property - It's useful to call out for
explicit testing here not necessarily for actually testing if the
property works (as we have a lot of automation that usually does
some of the testing here), but instead call out testing for web
compatibility concerns to find out if there are web compatibility
risks with unprefixing X DOM or CSS property.
How are we planning to test this? We have seen bugs in obscure web
sites which use the name of a new DOM property for example, but it seems
to me like there is no easy way for somebody to verify that adding such
a property doesn't break any popular website, even, since sometimes the
bug needs special interactions with the website to be triggered.
* Flagging of a risky patch needing to be sure it works - This is
where doing a deep dive generally by formulating a test plan and
executing it is useful.
This is sane too.
Note - I still think it's useful for a QA driver to look through a set
of bugs fixed for a certain Firefox release, it's just the process would
be re-purposed for flagging a bug for needing more extensive testing for
X purpose (e.g. web compatibility).
I'm not quite sure why that would be useful. If we believe that doing
blind verification is not helpful, why is doing that on a subset of bugs
fixed in a given release better?
Ehsan
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
