Hey Tony, Sorry we should have been more clear about that. We are using svn-dump (we aren’t doing this in real-time). Karanjeet can confirm, but from that dump file, I think we were able to generate this interactive app using DRAT in about a week or a little more. It’s worth noting that it’s *all* the revisions in that dump that we are scanning so this captures the evolution of the repo over time as well.
Thanks for the pointers to Git repos too. We’ll scan those with DRAT and Wrangler next. Also hat tip to the Solr community and so forth too - our DRAT stats are powered by Apache Solr (and D3). Cheers, Chris ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ -----Original Message----- From: Tony Stevenson <t...@pc-tony.com> Reply-To: "dev@community.apache.org" <dev@community.apache.org> Date: Wednesday, February 3, 2016 at 1:06 AM To: "dev@community.apache.org" <dev@community.apache.org>, Karanjeet Singh <karan...@usc.edu> Cc: "pierre.sm...@gmail.com" <pierre.sm...@gmail.com>, Infra <in...@apache.org> Subject: Re: DRAT is now scanning Apache SVN code base! >cc += infra@ > >Karanjeet, > >I am writing to you whilst wearing my Infrastructure hat. > >Please be careful if you are indeed recursing the entire ASF subversion >repository (http://svn.apache.org) - as you will quite likely run into >the aug-banning service. >Have you seen https://svn-dump.apache.org ? This is an entire dump of >the SVN repo (at least the public one you are interested in. You can use >this, and it is updated monthly. If you really need fully upto date data >you can use the dump, and svnsync the remaining revisions. > >I guess this might be obvious, but I’ll mention it just in case. A lot >of projects are using git repositories too. Which are mirrored here: >github.com/apache/ > > > >-- >Cheers, >Tony > >On behalf of the Apache Infrastructure Team > >----------------------- >http://www.pc-tony.com >GPG - 3072D/2543E323 >----------------------- > >> On 3 Feb 2016, at 08:58, Karanjeet Singh <karan...@usc.edu> wrote: >> >> Thanks Pierre for your feedback. >> >> Yes, the visualization corresponds to only 133 / 191 SVN projects ( >> http://svn.apache.org/repos/asf/). We have successfully audited close to >> 175 projects and hopefully by the end of this week all the remaining >> projects should be covered. We will update the data once done. >> >> Large repositories like "subversion" and "camel" having 493,420 files >>(size >> - 9,723 MB approx) and 519,584 files (size - 1,922 MB approx) taking up >>to >> 36 hours (only) to complete which is quite a good number. >> >> For your second question, I don't have an answer yet. Our intentions >>will >> be to update this regularly but we have some limitation at the Wrangler >>end >> that it doesn't allow us to run a job for more than 48 hours. Therefore, >> for very large repositories like openoffice, spamassassin, myfaces, etc, >> which takes more time to get audited, it will be a challenge to split >>the >> repositories every time and scan. >> >> Best Regards, >> Karanjeet Singh >> CS Graduate Student >> University of Southern California >> karan...@usc.edu | +1-213-675-9583 >> >> >> On Wed, Feb 3, 2016 at 12:06 AM, Pierre Smits <pierre.sm...@gmail.com> >> wrote: >> >>> HI Karanjeet, >>> >>> This is surely an impressive piece of work. But I still notice that >>>some >>> projects are missing in the overview. Is this a mere PoC not intended >>>to be >>> complete? Or something that will be made available to all and be >>>updated >>> regularly? >>> >>> Best regards, >>> >>> Pierre Smits >>> >>> ORRTIZ.COM < >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__www.orrtiz.com&d=CwI >>>BaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=u7neGGUaVmQKNSLUqJ9z >>>pA&m=I4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=rey8QvJVsx9VER8tfbyqc >>>WeBc3x1dze3BDFEgOry1zo&e= >>>> >>> OFBiz based solutions & services >>> >>> OFBiz Extensions Marketplace >>> >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__oem.ofbizci.net_oci- >>>2D2_&d=CwIBaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=u7neGGUaVm >>>QKNSLUqJ9zpA&m=I4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=t-3eq7_jE8P >>>3hTlTBYAQB9p_vFHuwoj6RqdbBBr8edI&e= >>> >>> On Wed, Feb 3, 2016 at 2:39 AM, Lewis John Mcgibbney < >>> lewis.mcgibb...@gmail.com> wrote: >>> >>>> Hi Karanjeet, >>>> >>>> A good bunch of work has lready gone into this and it is looking >>>>really >>>> friggin smart indeed. >>>> Interesting to see some many pieces of software come together and >>>>result >>> in >>>> something very easy to interpret. >>>> Good work. >>>> Lewis >>>> >>>> On Mon, Feb 1, 2016 at 11:44 PM, >>>><dev-digest-h...@community.apache.org> >>>> wrote: >>>> >>>>> Hello Everyone, >>>>> >>>>> With great pleasure, I would like to introduce DRAT (Distributed >>> Release >>>>> Audit Tool) which is a distributed, parallelized wrapper around >>>>>Apache >>>> RAT >>>>> to inspect for appropriate open source licensing in software >>>>>projects. >>>>> DRAT was started by my advisor, Chris Mattmann, in an effort to get >>>>>RAT >>>>> working on a ver large code base. RAT uses Apache OODT, Apache Tika, >>> and >>>>> Apache Solr. >>>>> >>>>> We are now auditing the complete Apache SVN code base to check for >>> proper >>>>> licenses. Until now, we have scanned 171 / 191 repositories and >>>>> illustrated the statistics for 133 of them through D3 visualization >>>>> located at >>> >>>https://urldefense.proofpoint.com/v2/url?u=http-3A__drat.dyndns.org-3A80 >>>80_dratviz&d=CwIBaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=u7ne >>>GGUaVmQKNSLUqJ9zpA&m=I4VmXy1BbrwbVZc9758zYzQ1Vg_gsve4ety_zu60Z7o&s=Eiqoi >>>xInVvAF49_1n7AxSu4q_q7BYMJ53JbVnf7rWK4&e= >>>>> >>>>> Projects should check out the MIME analysis of the code base and >>>>>click >>>>> around. Please also note due to the sheer size of the Apache code >>>>>bases >>>>> and the fact that we scanned and included all revisions in the Apache >>> SVN >>>>> repo, DRAT is not running in real time. We are running DRAT on the >>>>>NSF >>>>> Super Computer Wrangler, which has a petabyte of flash storage and >>>>>the >>>>> ability to stand up Hadoop and Spark clusters. We are also working >>>>>on a >>>>> paper describing our results. >>>>> >>>>> Please send feedback to myself (Karanjeet Singh <karan...@usc.edu>), >>>>> Professor Mattmann <mattm...@usc.edu> and/or >>> ird...@mymaillists.usc.edu. >>>>> >>>>> Thanks & Regards, >>>>> Karanjeet Singh >>>>> C.S. Graduate Student >>>>> University of Southern California >>>>> karan...@usc.edu | +1-213-675-9583 >>>> >>> >
