Top post with an example using the Virus Total API: > anteater --bincheck --project testproject --path /home/luke/repos/personal/anteater/tests/testproject
2018-02-13 14:49:18,349 - anteater.src.get_lists - INFO - Loaded testproject specific file_audits entries 2018-02-13 14:49:18,352 - anteater.src.get_lists - INFO - Loaded testproject specific file_contents entries 2018-02-13 14:49:18,375 - anteater.src.project_scan - INFO - Non Whitelisted Binary file: /home/luke/repos/personal/anteater/tests/testproject/images/pal.png 2018-02-13 14:49:18,376 - anteater.src.project_scan - INFO - Performing Scan: /home/luke/repos/personal/anteater/tests/testproject/images/pal.png 2018-02-13 14:49:18,824 - anteater.src.project_scan - INFO - File last scanned and shown as clean on:, 2018-02-13 13:44:11 2018-02-13 14:49:18,825 - anteater.src.project_scan - INFO - Full report here: https://www.virustotal.com/file/a71e13ebeb2500ed20781ab3ae8a9b306cf69a6c8be9a31e96d4e04f1657b4d8/analysis/1518529451 2018-02-13 14:49:18,825 - anteater.src.project_scan - INFO - The following sha256 hash can be used in your testproject.yaml file: a71e13ebeb2500ed20781ab3ae8a9b306cf69a6c8be9a31e96d4e04f1657b4d8 Should have the URL / Domain / IP stuff working later in the week. On Tue, Feb 13, 2018 at 9:41 AM, Luke Hinds <[email protected]> wrote: > > > On Tue, Feb 13, 2018 at 12:17 AM, SULLIVAN, BRYAN L (BRYAN L) < > [email protected]> wrote: > >> Comments etc inline >> >> >> >> Thanks, >> >> Bryan Sullivan | AT&T >> >> >> >> *From:* Luke Hinds [mailto:[email protected]] >> *Sent:* Monday, February 12, 2018 9:04 AM >> *To:* SULLIVAN, BRYAN L (BRYAN L) <[email protected]> >> *Cc:* [email protected]; degirmenci, fatih < >> [email protected]>; Raymond Paik <[email protected]> >> *Subject:* Re: [opnfv-tech-discuss] Anteater status and link issue >> >> >> >> >> >> >> >> On Tue, Feb 6, 2018 at 2:32 PM, SULLIVAN, BRYAN L (BRYAN L) < >> [email protected]> wrote: >> >> Hi all, >> >> I’m wondering where the Anteater program is – and want to note a broken >> link: build jobs with Anteater violations reference “Please visit: >> https://wiki.opnfv.org/x/5oey >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_x_5oey&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=s4zQHMsxrgVhlTs-Sw4-uGIsKYDMsnIQuvx0TehUoSk&e=> >> ”, which is the wiki page https://wiki.opnfv.org/pages/v >> iewpage.action?pageId=11700198 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_pages_viewpage.action-3FpageId-3D11700198&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=burTDZjfgUSG9lAKW4MjRDZULxleQEsKGknHvhdqzbA&e=>, >> which says “Project specific exceptions can be added for file_name, >> file_contents and binaries, by using the name of the repository within the >> anteater/exceptions/ directory of the releng-anteater >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_gerrit.opnfv.org-3A29418_releng-2Danteater.git&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=LrWykp0HOa_RUbxOEJDo7sojbPgNgsVsrlV6jmwMVx4&e=> >> repository.” – but that link (releng-anteater >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_gerrit.opnfv.org-3A29418_releng-2Danteater.git&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=LrWykp0HOa_RUbxOEJDo7sojbPgNgsVsrlV6jmwMVx4&e=>) >> is broken. >> >> I want to start adding the exceptions for Models etc as an example for >> the LF IT team that is setting up the Acumos project gerrit/CI/CD process, >> and in general to help optimize the Anteater overhead for projects. I think >> we need to get some analysis of the types of exceptions that are typical, >> and establish a process for vetting those exceptions that goes beyond a >> simple review by a releng committer. >> >> Further, we need to bring in other scan tools (e.g. security >> vulnerability, virus, or malicious code scans) into the Anteater process. >> This is in response to concerns about the security of the governance >> process for open source (e.g. upstream, but also direct contribution in >> projects) that is used to build production-oriented systems. We need to >> demonstrate that OPNFV and other LF projects are addressing these concerns >> through their infra toolsets. >> >> >> >> Sorry Bryan, I missed a few of these emails thanks (or rather no thanks) >> to a bad mail filter rule. >> >> I am working on the following now which we will see soon: >> >> Much better documentation: http://anteater.readthedocs.io/en/latest/ >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__anteater.readthedocs.io_en_latest_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=AdeEyIqajKWRGD1zz3MXcKrWoAWYR6mXmQDgVVzp1Zo&e=> >> >> [bryan] Are you going to start hosting these docs at docs.opnfv.org? >> > We can do yes, although I guess it make sense to have the main body of the > documentation around the tool upstream (once the github re-homing happens), > and then have everything OPNFV developers need to know about how anteater > is used in OPNFV at docs.opnfv.org - this way there won't be materials in > docs.opnfv.org around using Travis CI (which would confuse people). > >> Virus total integration: >> >> * Any binaries will be scanned using the virus total API, unless a >> sha256 waiver is already present e.g. https://github.com/opnfv/relen >> g-anteater/blob/master/exceptions/calipso.yaml#L9 >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_opnfv_releng-2Danteater_blob_master_exceptions_calipso.yaml-23L9&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=WNetEYMktH0pxwVzSJXZyDFVnJr6lIDBhM6laGBrbjs&e=> >> >> * Any IP addresses / domain name / URL will be scanned (again using >> the Virus Total API) for known malware and other nastiness. >> >> [bryan] VirusTotal looks like a useful service. Are there any stats for >> its effectiveness at detecting threats, including new threats and delay in >> supporting them? >> > Its pretty much the epicentre of community based threat collaboration . > It aggregates 40 virus / malware scanners to asses files, and domains / IP > addresses are assessed against 70 URL/domain blacklisting services: > > https://support.virustotal.com/hc/en-us/articles/115002126889-How-it-works > > It also fits well into anteater: > > Currently anteater will generate a sha256sum of any blobs it finds, and > will report them, *unless* a sha256 is entered into the exception files. I > will extend this, so that if a blob is found an *no* sha256sum exception > exists, we send the file hash to Virustotal to see if its registered as > nefarious. If it is we will fail the job and alarm the finding. If not, an > exception can be entered and it will be ignored from there on in and we > won't trouble the VT API again for that particular file - unless someone at > a later point changes the file (which would change the checkum) and then a > scan is made again - this way we can be sure that an infected file is not > checked into a project and we are not aware as it has the same name as > before. > >> I also have a load of new strings to add to dig out and report anything >> of a more recent finding (for example a javascript based bitcoin miner). >> >> [bryan] I would like to see how we can improve the contextual >> effectiveness of the pattern matching approach. Any bar (or port in a >> storm) may seem to be better than none, and can at least catch newbie >> mistakes and anti-patterns, but most of the strings I’ve included in >> https://github.com/opnfv/models/blob/master/tools/anteater-e >> xceptions.yaml relate to IMO innocuous (if admittedly sometimes cheap or >> anti-patterned) use of prohibited words. Others, I clearly need to fix. >> > > So I am very open to switching off the more noisy regexs that emit false > positives and also open to new approaches. I am sure I can fine tune them > much better as well. > > Likewise open to any feature recommendations etc. > >> The project is also hopefully going to move into github (once agreed with >> LF) to encourage wider contributions and allow it to be more easily >> consumed else. >> >> [bryan] Anything that broadens contribution and consumption makes sense >> to me. Are there any other open source projects in this same space that you >> are considering leveraging, to avoid re-developing features unnecessarily? >> > We plan to discuss wider LFN adoption , one example being OpenDayLight > where I manage security. I also plan to get more eyes on the tool for > smaller projects to utilise. An OpenStack project is also considering the > tool, but more for finding depreciated key directives and release tags. > >> Once the above is in place, docs will be clearer to follow, project will >> be more presentable, with more coverage in finding vulns will be wider. >> >> >> >> [bryan] We probably need more docs re the process for getting exceptions >> approved, and how the community can track its effectiveness in the mission >> represented by this toolset, through the types of approved exception >> patterns, as they grow (or shrink… it would be good to see the community >> improving through reduction in the need to maintain exceptions, and partly >> because the tool is getting smarter). >> > > Very much agree, it would be great to see people add to the master > exception / ignore list and feedback on where the tool works well / is > annoying etc. > > I also agree on the docs and enlarging upon the process for getting > exceptions approved. I plan to have all this done before ONS so we can see > it in place for then. > > Thanks, >> >> Bryan Sullivan | AT&T >> >> >> >> >> _______________________________________________ >> opnfv-tech-discuss mailing list >> [email protected] >> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss >> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.opnfv.org_mailman_listinfo_opnfv-2Dtech-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=8NPFgQFDZsv688HirOlM8HW1u0X9QVVgUfsN6B5PP_s&e=> >> >> -- Luke Hinds | NFV Partner Engineering | CTO Office | Red Hat e: [email protected] | irc: lhinds @freenode | t: +44 12 52 36 2483
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
