Thanks Mark. Please let us know how we can help make this fuzzing experience better for you. We're also happy to jump on a call to walk through your concerns and reach a good outcome.
Best regards, -- Oliver On Thu, 17 Nov 2022 at 06:56, Mark Thomas <ma...@apache.org> wrote: > I haven't forgotten about this. I am currently working through the open > issues. I want to complete first that so feedback isn't skewed by a > single project. > > Mark > > > On 11/11/2022 14:45, Roman Wagner wrote: > > Hi Mark, > > > > I think the best way forward is to collaborate and have a short feedback > > loop. > > > > Did you mean build failures by “Invalid due to broken test”? If yes, I am > > not sure what we can do about the broken tests since those are already > > executed and tested by check build scripts locally and in a CI/CD > pipeline. > > Build and Coverage failures are sometimes supposed to happen if there are > > changes in the target repository itself or if there are infrastructure > > issues in OSS-Fuzz. We will investigate those issues in more detail. > Maybe > > some filter in the apache mailing list is helpful for you in the short > > term, Fuzzing and Coverage build issues have a "build failure" string in > > the subject. That would enable you to focus on the reports only. > > > > In order to make sure that we get high-quality tests and results, > > maintainer feedback from apache will be very valuable and welcome. You > have > > the best domain knowledge about your code base and can give valuable > hints > > on which APIs to tackle best. There was already some valuable feedback > for > > Apache Tomcat in > https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53153. > > Let us extend this collaboration. We can discuss and agree on the attack > > vectors in apache-commons components. > > > > Best regards > > Roman > > > > On Thu, Nov 10, 2022 at 10:29 AM Mark Thomas <ma...@apache.org> wrote: > > > >> Oliver, > >> > >> My requirements regarding configuration are: > >> > >> - secur...@commons.apache.org MUST be notified of all security > >> vulnerability reports for all Apache Commons components > >> > >> - a mechanism MUST be provided for the secur...@commons.apache.org > >> Google user to view all historical reports that were not previously > >> notified to that address > >> > >> - if any further Apache Commons components get added to oss-fuzz > >> then secur...@commons.apache.org MUST receive notification of any > >> issues as they are found > >> > >> - more generally, if *any* Apache Software Foundation project is added > >> to oss-fuzz then the notifications for that project MUST include the > >> relevant security team for that project > >> > >> If you can achieve the above with the current structure then great. > >> > >> > >> Separately, there is a concern regarding the false positive rate. With > >> the oss-fuzz integration provided by Code Intelligence we have seen the > >> following with Apache Tomcat in a little under 3 months. > >> > >> Total "vulnerability" reports: 39 > >> > >> Invalid due to broken test: 31% > >> False positive: 52% > >> Bugs: 18% > >> Valid security issues: 0% > >> > >> To add some commentary: > >> - the bugs were minor / extreme edge cases users were unlikely to hit > >> - false positives were all due to the tests being based on invalid > >> assumptions regarding whether input was expected to be trusted or > not > >> > >> > >> If those statistics were repeated across multiple Apache Commons > >> components, the volume of invalid reports would be more than the > >> volunteers of the Apache Commons security team could handle. > >> > >> I have no objection to being overwhelmed with valid security > >> vulnerability reports. If that ever happened, we would find a way to > >> deal with it. > >> > >> I do have very strong objections to being overwhelmed with invalid > >> security vulnerability reports. If we see the same false positive rate > >> repeated across the Apache Commons components that has been observed > >> with Apache Tomcat then I don't see that Apache Commons would have any > >> choice but to request the removal of all Apache Commons Components from > >> oss-fuzz. > >> > >> Mark > >> > >> > >> > >> > >> > >> On 10/11/2022 04:19, Oliver Chang wrote: > >>> Hi Mark, > >>> > >>> In addition to the reasons Roman listed, the current structure also > >>> allows us to allocate more compute resources to all of these Apache > >>> packages, rather than all of them sharing the CPUs allocated for a > >>> single OSS-Fuzz "project". > >>> > >>> We can definitely ensure that secur...@commons.apache.org > >>> <mailto:secur...@commons.apache.org> is included on all relevant > Apache > >>> projects going forward, and other than that I believe there's not much > >>> other difference in terms of the end result (i.e. bug reports) that end > >>> up getting filed. > >>> > >>> Does that sound OK to you? Or did you have other concerns around the > >>> current directory structure? > >>> > >>> Best regards, > >>> -- > >>> Oliver > >>> > >>> > >>> On Wed, 9 Nov 2022 at 21:31, Roman Wagner < > wag...@code-intelligence.com > >>> <mailto:wag...@code-intelligence.com>> wrote: > >>> > >>> Hi Mark, > >>> > >>> I have added @Oliver Chang <mailto:och...@google.com> from the > >>> Google OSS-Fuzz to the thread. > >>> > >>> I had a short discussion with Oliver. There could be different > >>> issues in OSS-Fuzz by design If all apache-commons components will > >>> move under apache-commons directory: > >>> > >>> * it is not scalable and will slow down both fuzzing and triage > >>> (e.g. automated bisections, fix verification) > >>> * changing the structure this way will invalidate all existing > >>> open testcases, and cause new ones to be filed, which will > >>> result in a fair bit of spam. > >>> > >>> My proposal would be that "secur...@commons.apache.org > >>> <mailto:secur...@commons.apache.org>" is added to all individual > >>> apache-commons components. > >>> I am not sure how it is possible to ensure that future onboardings > >>> of apache-commons components will automatically have > >>> "secur...@commons.apache.org <mailto:secur...@commons.apache.org > >" > >>> as primary contact. OSS-Fuzz could have some additional > >>> documentation for that. @Oliver Chang <mailto:och...@google.com> > do > >>> you have any ideas here? > >>> > >>> Best regards > >>> Roman > >>> > >>> On Tue, Nov 8, 2022 at 5:56 PM Mark Thomas <ma...@apache.org > >>> <mailto:ma...@apache.org>> wrote: > >>> > >>> Thanks for the update. > >>> > >>> I'll wait for that PR to be resolved before taking any further > >>> action. > >>> > >>> Mark > >>> > >>> > >>> On 08/11/2022 16:42, Roman Wagner wrote: > >>> > Hi Mark, > >>> > > >>> > there is a PR open in oss-fuzz > >>> https://github.com/google/oss-fuzz/pull/8933 > >>> <https://github.com/google/oss-fuzz/pull/8933> > >>> > . > >>> > > >>> > Best regards > >>> > Roman > >>> > > >>> > On Tue, Nov 8, 2022 at 4:15 PM Gary Gregory > >>> <garydgreg...@gmail.com <mailto:garydgreg...@gmail.com>> > wrote: > >>> > > >>> >> Sounds good. > >>> >> > >>> >> Gary > >>> >> > >>> >> On Tue, Nov 8, 2022, 10:07 Mark Thomas <ma...@apache.org > >>> <mailto:ma...@apache.org>> wrote: > >>> >> > >>> >>> There has been no response to this email from anyone from > >> Code > >>> >>> Intelligence. > >>> >>> > >>> >>> Unless there are objections from the Apache Commons > >>> Community my next > >>> >>> step will be to submit a PR to have the following modules > >>> removed from > >>> >>> oss-fuzz: > >>> >>> > >>> >>> apache-commons-bcel > >>> >>> apache-commons-beanutils > >>> >>> apache-commons-cli > >>> >>> apache-commons-codec > >>> >>> apache-commons-collections > >>> >>> apache-commons-configuration > >>> >>> apache-commons-io > >>> >>> apache-commons-jxpath > >>> >>> apache-commons-lang > >>> >>> apache-commons-logging > >>> >>> > >>> >>> Code Intelligence (or anyone else) will remain free to > add > >>> them back in > >>> >>> the right place - under apache-commons should they wish > to > >>> do so. > >>> >>> > >>> >>> Mark > >>> >>> > >>> >>> > >>> >>> > >>> >>> On 19/10/2022 10:56, Mark Thomas wrote: > >>> >>>> Hi, > >>> >>>> > >>> >>>> You are receiving this email as you are currently > >>> configured as the > >>> >>>> recipients for oss-fuzz reports for Apache Commons > JXPath. > >>> >>>> > >>> >>>> As per the discussion on the Apache Commons dev list[1], > >>> please make > >>> >>> the > >>> >>>> following configuration changes to the oss-fuzz > >>> integrations with > >>> >>>> immediate effect: > >>> >>>> > >>> >>>> - Move all oss-fuzz integrations added for *ALL* Apache > >>> Commons > >>> >>>> components to the oss-fuzz module for > Apache-Commons: > >>> >>>> > >>> >>>> > >>> >>> > >>> > >> https://github.com/google/oss-fuzz/tree/master/projects/apache-commons > < > >> https://github.com/google/oss-fuzz/tree/master/projects/apache-commons> > >>> >>>> > >>> >>>> There should *NOT* be separate oss-fuzz modules for > >>> each component > >>> >>>> > >>> >>>> > >>> >>>> - Add the Google account for " > secur...@commons.apache.org > >>> <mailto:secur...@commons.apache.org>" to > >>> >>>> - the notifications for these issues > >>> >>>> - the ACL to enable this account to access the > details > >>> for each > >>> >>> report > >>> >>>> > >>> >>>> > >>> >>>> Please notify dev@commons.apache.org > >>> <mailto:dev@commons.apache.org> and > secur...@commons.apache.org > >>> <mailto:secur...@commons.apache.org> > >>> >>>> when these changes have been completed. > >>> >>>> > >>> >>>> Thanks, > >>> >>>> > >>> >>>> Mark > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> [1] > >>> > https://lists.apache.org/thread/53vwy3g8w3f8nydz7jvxm8snrqx7msln > >>> < > >> https://lists.apache.org/thread/53vwy3g8w3f8nydz7jvxm8snrqx7msln> > >>> >>>> > >>> >>>> > >>> > >> --------------------------------------------------------------------- > >>> >>>> To unsubscribe, e-mail: > dev-unsubscr...@commons.apache.org > >>> <mailto:dev-unsubscr...@commons.apache.org> > >>> >>>> For additional commands, e-mail: > >>> dev-h...@commons.apache.org <mailto: > dev-h...@commons.apache.org> > >>> >>>> > >>> >>> > >>> >>> > >>> > >> --------------------------------------------------------------------- > >>> >>> To unsubscribe, e-mail: > dev-unsubscr...@commons.apache.org > >>> <mailto:dev-unsubscr...@commons.apache.org> > >>> >>> For additional commands, e-mail: > >>> dev-h...@commons.apache.org <mailto: > dev-h...@commons.apache.org> > >>> >>> > >>> >>> > >>> > > >>> > >>> > >> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > >>> <mailto:dev-unsubscr...@commons.apache.org> > >>> For additional commands, e-mail: dev-h...@commons.apache.org > >>> <mailto:dev-h...@commons.apache.org> > >>> > >>> > >>> > >>> -- > >>> > >>> Roman Wagner > >>> Application Security Engineer > >>> > >>> Code Intelligence > >>> Rheinwerkallee 6 > >>> 53227 Bonn > >>> > >>> Amtsgericht Bonn > >>> HRB 23408 > >>> > >>> Geschäftsführer: Sergej Dechand, Dr. Khaled Yakdan > >>> > >> > > > > >
