On 1/29/2016 6:45 PM, Emma Humphries wrote:
Why This Matters
Bugs are a unit of quality we can use to see how we’re doing.
We believe that the number of outstanding, actionable bugs is the best
metric of code quality available, and that how this number changes over
time will be a strong indicator of the evolving quality of Firefox.
Actionable bugs are those with the status NEW for which the next action
required - if any - is unambiguous: does the bug need immediate attention,
will it be worked on for a future release, will it be worked on at all?
As the sponsor for this project, I want to clarify some things related
to this project. The bug-handling initiative is a key part of our
Firefox quality programs in 2016. I have asked Emma to focus on our
process for making explicit decisions about incoming bugs. There are two
primary reasons for this:
* When we don't see incoming regression bugs, that is high risk to our
quality and shipping schedule. We have done dot-releases or similar
respins numerous times over the past year: in many cases the bugs that
caused those respins were filed well before release but either not seen
or we didn't react to them.
* Our testing community, especially prerelease users, are a core part of
our success. We know that one of the greatest turn-offs for people
filing their first bug is to have it sit with no response. On the other
hand, a quick response is a good predictor of continued and deeper
involvement in the project.
I don't think that we are going to learn much about quality or risk just
by counting the number of open bugs. So I don't think the statement
"best metric of code quality available" is true, and it has generated a
lot of the feedback in this thread. But I do assert that a core risk
metric is the number of new bugs without a *decision*, and we should be
tracking that number and driving it to zero.
Part of the program that is already underway is triaging everything out
of Core:Untriaged and Firefox:Untriaged into the proper component. This
is a task that we've been accomplishing both with our volunteer
community and with the support of Softvision contractors, and we're on
track to burn the Untriaged components to 0 bugs by the end of this quarter.
My intention, once we have this system in place, is to focus activities
on increasing the both the quality and quantity of incoming bug reports,
especially from our prerelease users: building product features which
enable users to file more detailed and useful bug reports, combined with
data collected within the browser. Filing a bug in bugzilla is still a
scary experience for many people, and we can do better.
To keep focus and avoid creeping scope, an explicit non-goal of this
program is to deal with the prioritization of non-critical bugs within a
team or component. The primary goal here is to solve the flow for
incoming bugs to a clear decision-state. Beyond the bucket of "backlog
of work", teams can continue to use aha or tracking bugs or external
spreadsheets.
One other important thing to note is that we plan on implementing
bugzilla UI improvements to make triage much simpler. This may include
things like one-click decision making on bugs and autoloading bug
reports from triage lists. We expect that work to commence once the
initial trials are done in Q1 and we have better experience with the
details.
To reply to a few specific comments and questions...
Richard Newman wrote:
In my experience, component watching adequately serves this purpose,
and component watchers collaboratively respond to, immediately triage,
or flag with a tracking-? flag. That might not be true for some
components, but it is for the three products I watch.
I disagree that component watching is working well now. There are
components that are well-watched, and it mostly works to surface
high-priority (tracking+) bugs. But for non-critical bugs, they are
often left filed as NEW without even so much as a comment explaining a
decision: the bug reporter is left without a clear understanding of what
to expect or even knowledge that somebody has looked at their bug
report. We owe it to our community of bug filers to be more clear and
explicit with these decisions.
I think it's worth pointing out that it's not always worth assigning
bugs to a developer — in my experience that does more harm than good
if they're not working on it. Be realistic. Neither does P1 mean
anything useful in many of the components I follow.
Emma is talking explicitly about bugs which rise to the level of a
tracking+ bug (critical regression or a bug in a new feature). Right now
we're living in a world where release drivers mark a bug tracking+ bug
many of these bugs remained assigned to nobody forever. This is a bad
state and the goal is to start sending out daily alerts for tracking+
bugs without an owner.
Many of our regressions are reported informally on Twitter, as GitHub
issues, or on IRC. Those are spotted by engineers or QA who go
looking, and those people file bugs. Those bugs, naturally, enter the
funnel half-way along, skipping the pre-triage you propose. Can that
be improved or leveraged?
The goal is that no bug will be able to skip the per-component
decision-making process. Emma will be experimenting with early groups to
figure out how to make this work.
We have "tracking-+", "P2", etc. bugs that will realistically never be
addressed, because there's always more important work to do. What can
we do to not lose sight of those and adjust priority accordingly?
I don't know about P2 in this context, but I don't think that we can
afford to leave tracking+ bugs unassigned or unaddressed. That is an
essential part of our core quality focus. If a team is only working on
tracking+ bugs and is still understaffed, that is something we need to
be aware of and fix.
Gervase Markham wrote:
Without wanting to pile on, as I know others have concerns about other
parts of this plan, and without wanting to say it's only you doing this,
but: can we all please stop using the word "community", as this sentence
implies, as "people outside the paid 'team' who get to work on things
which are not important enough for the important people to spend their
time on"? Community is not the antonym of "team", nor is it the antonym
of "employee".
This wording was intentional and we thought carefully about it. The
"team" in this case includes both employees and volunteers who are
working together from a shared list of priorities, and is responsible
for a particular area of code and bugzilla component. That team can make
a decision that they, as a group, are not going to prioritize a
particular bug ever, but still accept patches from others outside the
team (employees or volunteers!) who want to fix it.
Marco Bonardo wrote:
Most of this plan assumes every component has a team that works on it
(let alone triage meeting). Well, that's not always the case, some of
the components have a responsible that is working on something else or
part of another team.
For example I'm responsible for Places, I watch the component and try
to keep it in order as far as possible, but there's no team and it's
not my primary job at the moment. This is an example but it's valid
for a bunch of other components, especially in Firefox where the teams
cross various components depending on the goal.
This is both true, and part of the problem. We cannot claim that we have
a focus on quality if we don't even have the resources to look at the
incoming bugs. The goal is to make daily bug triage very fast and painless.
That said, the incoming bug rate for Places is very low: in Q4 2015
there were a total of 30 bugs filed or triaged into Toolkit:Places:
https://bugzilla.mozilla.org/buglist.cgi?list_id=12831181&chfieldto=2015-12-31&query_format=advanced&chfield=[Bug%20creation]&chfieldfrom=2015-09-01&component=Places
Dave Townsend wrote:
I'm concerned about making this sort of demand when the component
owner is not an employee, or is the expectation that only employees
would be in these roles?
It doesn't have to be a single person. Maybe it means a rotation with a
different person each day, or a weeklong rotation. Mike Hoye is working
on a program this quarter to build component triage leads from the
community. In the past, this has been one of our big success points in
developing the community. In the past I have had triage leads for
plugins who were much more knowledgeable than I was about the state of
the various trees. There is also I think a natural progression from
component triage lead to a role within product management that we need
to explore.
--BDS
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
