On Wed, Mar 9, 2016 at 12:58 PM, David Rajchenbach-Teller < dtel...@mozilla.com> wrote:
> I'm assuming that, for the sake of this discussion, Gecko == > mozilla-central minus stuff that can be built and distributed on its > own. In particular, most of toolkit/ is covered by this definition. > > Just looking at things I use/work on, I believe that we could relatively > easily split in distinct modules, with distinct repos and separate > fuzzing/testing: > - the crash reporter; > - the crash manager; > - the shutdown terminator; > - AsyncShutdown; > - Telemetry; > - OS.File; > - Task.jsm. > Modularizing these things would be great, but I don't see anything on this list that I think we would realistically fuzz given our current resourcing. > > > Certainly, this is just a few ‱ of Firefox, but, hey, you need to start > somewhere. I believe that chipping away at the nightmare of implicit > dependencies that Gecko has become would be a good idea. Fwiw, 90% of my > work on (Async)Shutdown is finding out which implicit and undocumented > hypothesis on execution order have been introduced in the code and > shaving as much yak as needed to make them explicit, well-behaved and > not-hangy/crashy. > > If splitting Gecko Cargo-style, across repos with clear dependencies, is > the best way to do this, I'm happy to head that way. > I think splitting Gecko into multiple repos is an anti-goal, because we derive enormous productivity benefits from using a mono-repo. If you're serious about proposing that, I would suggest starting a new thread to avoid derailing this one. > More generally, I have recently come to realize how scared I am of my > own code, in particular JS-based. I know how it is supposed to behave, > but I have no idea how it is going to behave if, say, someone has the > brilliant idea of throwing an Error that cannot be converted to a String > (yes, this has happened to me, much fun ensued), or stopping a process > while I'm communicating to it, etc. > > If someone finds a way to introduce fuzz-testing that can simulate all > of that, I'm all for it. > > > Cheers, > David > > On 09/03/16 19:14, Bobby Holley wrote: > > Can you elaborate on which Gecko components you're hoping to fuzz > > separately? A lot of the core is pretty heavily-intertwined, so I'm > pretty > > skeptical that we'd ever be able to separate out DOM, style, and layout > > from each other (for example). There are basically two barriers: > > > > (1) These components are enormous, and were not built to be very modular. > > Any such efforts would require a huge amount of engineering resources, > > which we would probably not spend just to make the component fuzzable. > > (2) The performance cost of adding an abstraction layer between > > tightly-coupled components in C++11 would probably be prohibitive (the > > situation is different for Rust/Servo because of Traits - we could > > potentially do this with C++ Concepts/Modules in around a decade). > > > > This is all to say that I think a general call to "modularize Gecko" > isn't > > really helpful. But if there are specific leaf-y components that you want > > to fuzz separately but can't, that might be a good starting point. > > > > On Wed, Mar 9, 2016 at 9:54 AM, <twsm...@mozilla.com> wrote: > > > >> On Wednesday, March 9, 2016 at 9:38:55 AM UTC-8, Nicolas B. Pierron > wrote: > >>> This discussion is a follow-up discussion to some emails sent privately > >> by > >>> accident. > >>> > >>> If you have not followed, I will quote David Bryant: > >>> > Improving release quality is one of the three fundamental goals > >> Platform > >>> > Engineering committed to this year. To this end, lmandel built a > >> Bugzilla > >>> > dashboard that allows us to track regressions found in any given > >> release > >>> > cycle. This dashboard [...] can > >>> > also be found at: http://mozilla.github.io/releasehealth/ > >>> > >>> To David's email, I answered the following: > >>> > >>> ---------- > >>> tl;dr: If we want to improve the quality of our products we should > >>> split Gecko in standalone programs which are fuzzing-friendly. > >>> > >>> One thing which strikes me, is the ratio of regressions per component > >>> that we have for each versions, and more over who are the persons > >>> opening these bugs: > >>> - Release: > >>> > >> > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > >>> - Beta: > >>> > >> > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > >>> - Aurora: > >>> > >> > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > >>> - Nightly: > >>> > >> > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > >>> > >>> To be more precise: > >>> - The small number of regression we have in the JS engine on the > >>> release channel, versus the Extremely Huge number of regressions we > >>> have on nightly. > >>> - And the fact that (almost) all the bugs opened against the JS > >>> engine are opened by our fuzzing team. > >>> > >>> What I want to remark is the fact that our automated fuzzing is better > >>> at finding recently introduced regressions. And as far as I know, > >>> Alice is not a bot. > >>> > >>> From what I know, the reason fuzzing team is so efficient on the JS > >>> engine is because we have a *standalone* JS shell. > >>> The *standalone* JS shell is also the reason why our build time is > >>> below 2 minutes as opposed to 18 minutes. > >>> > >>> So, I think that if we want to improve our quality we should focus on > >>> making fuzzing-friendly standalone programs for the different > >>> components of the platform. > >>> Thus reducing, the compilation time, reducing the test suite time, and > >>> improving the ability of the fuzzing team to find recently added > >>> regressions. > >>> > >>> Maybe I am wrong, in which case the other alternative might be to > >>> staff the JS Team to get rid of all these nightly issues before they > >>> ride the train to release. > >>> ---------- > >>> > >>> To which I got the following replies: > >>> > >>> On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote: > >>> > The ratio of engineers to code in the js engine is so much higher > than > >>> > the rest of the product that I'm not sure this is a sensible > >> comparison. > >>> > The js engine also doesn't depend on things like 3rd party gfx > >> drivers ... > >>> > >>> On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote: > >>> > Fuzzing captures only a fraction of issues. > >>> > >>> On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote: > >>> > On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote: > >>> >> > >>> >> The ratio of engineers to code in the js engine is so much higher > >> than the > >>> >> rest of the product that I'm not sure this is a sensible > >> comparison. The js > >>> >> engine also doesn't depend on things like 3rd party gfx drivers ... > >>> > > >>> > This is probably not the only step that we need to take to > >> substantially > >>> > improve quality so setting up a place to have those discussions is > >> good. It > >>> > really is worth some time and effort to brainstorm about all the > >> things we > >>> > might do to raise the bar, poke some holes in those ideas, then > >> decide on > >>> > and push forward on a few more in the next few quarters. > >>> > >>> On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote: > >>> > On 3/9/16 6:58 AM, Nicolas B. Pierron wrote: > >>> >> So, I think that if we want to improve our quality we should focus > on > >>> >> making fuzzing-friendly standalone programs for the different > >>> >> components of the platform. > >>> >> Thus reducing, the compilation time, reducing the test suite time, > >> and > >>> >> improving the ability of the fuzzing team to find recently added > >>> >> regressions. > >>> > The fuzzing team has asked for this from different teams for various > >>> > components, with different degrees of support and resistance, > >> depending > >>> > on the component and team. > >>> > > >>> > It is work to do this but it changes the fuzzing efficiency and > >> ability > >>> > to find focused issues when we have this level of support, such as > we > >>> > have with the JS shell. It allows the team to fuzz directly on a > >>> > component and not have to find a way to reach it through Firefox and > >> all > >>> > the additional noise/assertions/etc that it churns up. > >>> > >>> -- > >>> Nicolas B. Pierron > >> > >> I think Nicolas is right on the mark! JS shell is a good example of > using > >> a shell for fuzzing and I think media and graphics could also get a good > >> deal out of a shell. > >> > >> To get a bit more specific breaking things in to smaller pieces for > >> fuzzing give us (the fuzzing team) a major advantage. In addition to > what > >> Nicolas mentioned, it makes it much easier to use different sanitizers > >> (Address, Memory, Undefined behavior, etc...) It also makes it possible > to > >> use coverage guided fuzzing which is a major advancement in fuzzing. I > have > >> been doing a lot of work fuzzing 3rd party libraries that we use in > Firefox > >> and there have been many times I wished I could have use the same tools > as > >> easily on our own code. That said the fuzzing on the browser as a whole > it > >> also important as well. > >> > >> More dev support for the fuzzing means more bugs found and sooner! > >> > >> _______________________________________________ > >> dev-platform mailing list > >> dev-platform@lists.mozilla.org > >> https://lists.mozilla.org/listinfo/dev-platform > >> > > _______________________________________________ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
