On Wednesday, March 9, 2016 at 9:38:55 AM UTC-8, Nicolas B. Pierron wrote: > This discussion is a follow-up discussion to some emails sent privately by > accident. > > If you have not followed, I will quote David Bryant: > > Improving release quality is one of the three fundamental goals Platform > > Engineering committed to this year. To this end, lmandel built a Bugzilla > > dashboard that allows us to track regressions found in any given release > > cycle. This dashboard [...] can > > also be found at: http://mozilla.github.io/releasehealth/ > > To David's email, I answered the following: > > ---------- > tl;dr: If we want to improve the quality of our products we should > split Gecko in standalone programs which are fuzzing-friendly. > > One thing which strikes me, is the ratio of regressions per component > that we have for each versions, and more over who are the persons > opening these bugs: > - Release: > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > - Beta: > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > - Aurora: > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > - Nightly: > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > To be more precise: > - The small number of regression we have in the JS engine on the > release channel, versus the Extremely Huge number of regressions we > have on nightly. > - And the fact that (almost) all the bugs opened against the JS > engine are opened by our fuzzing team. > > What I want to remark is the fact that our automated fuzzing is better > at finding recently introduced regressions. And as far as I know, > Alice is not a bot. > > From what I know, the reason fuzzing team is so efficient on the JS > engine is because we have a *standalone* JS shell. > The *standalone* JS shell is also the reason why our build time is > below 2 minutes as opposed to 18 minutes. > > So, I think that if we want to improve our quality we should focus on > making fuzzing-friendly standalone programs for the different > components of the platform. > Thus reducing, the compilation time, reducing the test suite time, and > improving the ability of the fuzzing team to find recently added > regressions. > > Maybe I am wrong, in which case the other alternative might be to > staff the JS Team to get rid of all these nightly issues before they > ride the train to release. > ---------- > > To which I got the following replies: > > On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote: > > The ratio of engineers to code in the js engine is so much higher than > > the rest of the product that I'm not sure this is a sensible comparison. > > The js engine also doesn't depend on things like 3rd party gfx drivers ... > > On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote: > > Fuzzing captures only a fraction of issues. > > On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote: > > On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote: > >> > >> The ratio of engineers to code in the js engine is so much higher than the > >> rest of the product that I'm not sure this is a sensible comparison. The > js > >> engine also doesn't depend on things like 3rd party gfx drivers ... > > > > This is probably not the only step that we need to take to substantially > > improve quality so setting up a place to have those discussions is good. > It > > really is worth some time and effort to brainstorm about all the things we > > might do to raise the bar, poke some holes in those ideas, then decide on > > and push forward on a few more in the next few quarters. > > On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote: > > On 3/9/16 6:58 AM, Nicolas B. Pierron wrote: > >> So, I think that if we want to improve our quality we should focus on > >> making fuzzing-friendly standalone programs for the different > >> components of the platform. > >> Thus reducing, the compilation time, reducing the test suite time, and > >> improving the ability of the fuzzing team to find recently added > >> regressions. > > The fuzzing team has asked for this from different teams for various > > components, with different degrees of support and resistance, depending > > on the component and team. > > > > It is work to do this but it changes the fuzzing efficiency and ability > > to find focused issues when we have this level of support, such as we > > have with the JS shell. It allows the team to fuzz directly on a > > component and not have to find a way to reach it through Firefox and all > > the additional noise/assertions/etc that it churns up. > > -- > Nicolas B. Pierron
I think Nicolas is right on the mark! JS shell is a good example of using a shell for fuzzing and I think media and graphics could also get a good deal out of a shell. To get a bit more specific breaking things in to smaller pieces for fuzzing give us (the fuzzing team) a major advantage. In addition to what Nicolas mentioned, it makes it much easier to use different sanitizers (Address, Memory, Undefined behavior, etc...) It also makes it possible to use coverage guided fuzzing which is a major advancement in fuzzing. I have been doing a lot of work fuzzing 3rd party libraries that we use in Firefox and there have been many times I wished I could have use the same tools as easily on our own code. That said the fuzzing on the browser as a whole it also important as well. More dev support for the fuzzing means more bugs found and sooner! _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
