On Mon, Jun 18, 2018 at 7:15 AM, Theodore Y. Ts'o <ty...@mit.edu> wrote: > I'd like to commend to you the style of bug reports which Wen Xu, > researcher at Georgia Tech has been using. (He's a Ph.D. student, > with an interesting in fuzzing, and he's been very responsive to my > suggestions about how to make his fuzzing much more impactful.) > > Please see some of his bugs: > > https://bugzilla.kernel.org/buglist.cgi?bug_status=RESOLVED&component=ext4&email1=wen.xu%40gatech.edu&emailreporter1=1&emailtype1=substring&query_format=advanced&resolution=CODE_FIX > > https://bugzilla.kernel.org/buglist.cgi?bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&component=ext4&email1=wen.xu%40gatech.edu&emailreporter1=1&emailtype1=substring&query_format=advanced > > As of this writing, I've found and fixed 13 bugs found by his fuzzing > system, out of 26 bugs that he filed with the kernel Bugzilla (the > others ended up being duplicates of the other reports). Of special > note that is the fact that he submits a separate image file from the C > file used to provoke the failure. > > He has also been submitting cut-down reproducer C files, and more > recently he has been creating cut-down fuzzed images. This is > currently been done by hand, but I've been suggesting that finding > ways to do this automatically might be a great area for future > research. > > In any case, his methods, which do require a bit more manual effort on > his part at the moment, save *me* as the developer a huge amount of > effort on my part. And so I really appreciate his method of engaging > with file system developers. > > Creating a smaller number of actionable, impactful reports, is in my > opinion far more productive than spamming developers with reports that > are either not actionable (due to the lack of a reproducer) or > extremely difficult to act upon, given the extremely limited amount of > resources developers have available to respond to these sorts of > reports. > > (Most of my work responding to fuzzing reports has been done at night > or on the weekends, and it's not something my employer pays me to work > on. So help from the bug reporters, which otherwise would vastly > outnumber me, is greatly appreciated. Spamming me with not > actionable, or hard-to-act-upon bug reports, simply doesn't scale.)
+LKML Hi Ted, I appreciate your feedback but please take into account significant difference in context and goals of what they are doing and what we are doing: 1. 26 bugs vs 2600 bugs. 2. Specialized testing for a single subsystem vs coverage of a hundred of kernel subsystems. 3. Personal interest in getting a dozen of trophies to publish research and forget about it vs setting a continuous lasting process for systematic bug reporting. What we are trying to do can't work without active interest and participation of all of kernel development community, without thousands of kernel developers who know, write and maintain all these subsystems. While we do have interest in all of these bugs getting fixed, we don't have specific personal interest in any single bug or any single subsystem. This needs to be completed by more specific interest in each particular subsystem from maintainers of that subsystem. In this context it's much more reasonable if fs maintainers cut down fs images, USB maintainers cut down USB packets and we continue improving tooling. Rather than we cut down images, you cut down USB packets, Alan Stern cuts down network packets, and Eric Dumazet improves tooling. When we reported bugs manually we indeed did more manual work on our side in some cases. But the problems were: (1) we couldn't keep up reporting all bugs even making this our full time job, (2) bugs were lost on kernel mailing lists in massive amounts (which is frustrating when it's your manual work), (3) while some bugs got more "personal touch", others got even less, sometimes we even mess or forgot details, (4) this frequently lead to wasted work when you spent half day getting everything perfect, but then you get "nah, known issue", or "oh, I just looked into code for 10 secs and I see the problem", or, well, no response at all, this is quite unpleasant experience. And in the end this means no automation, no improvements to tooling and no growth of kernel test coverage as all our time is already taken. In short, it did not work. One could say that then it's better to report few bugs but with full manual assistance on the reporter side. I tend to disagree: 1. At the very least we need to fix bugs faster than we introduce them. And with a single person as a bottleneck this is not happening. There are thousands of people introducing new bugs. 2. Who will do all of the uninteresting, mechanical, assistive work? Any volunteers? I don't think there will be any. And I don't even think it's a healthy thing for a software project to try to push all this work onto a dedicated group of people. 3. You say "not actionable reports without reproducers", but you can find hundreds of fixed bugs without reproducers at [1] and [2]. Fix ratio for bugs without reproducers is 66% which is not significantly lower than 76% for bugs with reproducers. A person without expertise in a particular subsystem (me) can't know if a bug is actionable by an expert in the subsystem (you) without first reporting this bug. 4. Reporting any bug is not a negative thing. First of all, it may be just very easy to fix (but we will never know until it's reported). And if nobody has time to look into it deeper, or it's really unactionable at this point, well, ok, being aware of state of the things is still better than being blind. Over time additional information may pop up, or several developers may contribute pieces of insight which will then lead to a fix (we are seeing such cases too). Looking at the data, reporting all bugs is clearly having overall positive effect. I mean a particular developer can receive a particular report that they think is non-actionable, and they are frustrated by the "spam". But let's not consider this in isolation. Realistically there is no option of somehow retroactively report only bugs that will be well accepted, right. Please look at this in larger context. Over time quality of reports will improve and that's on us (though, syzkaller is also an open-source project and contributions are welcome). To conclude, we need active involvement of all of kernel community to improve the situation. There are still plenty of unfixed bugs [3]. An expert in a subsystem may be able to quickly identify that this single thing in a reproducer is the culprit; or that there were recent changes in that part of code so that's the likely culprit; or just seeing the problem in code right away. Our team is really a negligible part of the overall equation. [1] https://syzkaller.appspot.com/?fixed=upstream [2] https://github.com/google/syzkaller/blob/master/docs/linux/found_bugs.md [3] https://syzkaller.appspot.com/
