I set this up and configured Jenkins to use the test-data checked out from Subversion.
Uwe ----- Uwe Schindler uschind...@apache.org ASF Member, Apache Lucene PMC / Committer Bremen, Germany http://lucene.apache.org/ > From: Steve Rowe [mailto:sar...@gmail.com] > Sent: Monday, August 15, 2016 8:40 PM > To: d...@lucene.apache.org > Subject: Re: JDK, Maven, Ant versions have been consolidated. > > Thanks Uwe for moving this process along. > > +1 to make a new SVN dir at https://svn.apache.org/repos/asf/lucene/test- > data/ and put the enwiki data file there. > > -- > Steve > www.lucidworks.com > > > On Aug 15, 2016, at 1:47 PM, Uwe Schindler <uschind...@apache.org> > wrote: > > > > Hi, > > > > I thought about the wikipedia test files used by Jenkins: I think I would > commit them to SVN inside the Lucene/Solr project’s folder. The Nightly-Test > Jenkins jobs that use them could simply check them out into a separate dir of > the workspace. Byt that it is easier for us to update them. If you also think > this is fine, we can leave it like that. > > > > The ~jenkins/lucene.build.properties file is (as said before) > machine/hardware specific. I’d keep them in Jenkins’ home dir. We may > create a puppet module out of it, but this would prevent us from optimizing > or changing them quickly per project requirements. > > > > Finally the “ant ivy-bootstrap” task was for now separated into a Jenkins > job, that can be manually run (https://builds.apache.org/job/Lucene-Ivy- > Bootstrap/). In the future, we can add the “ivy-bootstrap” target as > dependency to the “ant jenkins” and other targets which are solely triggered > by Jenkins, so separately bootstrapping is no longer required (please keep in > mind that bootstrapping places the ivy.jar file in ~/.ant/lib, so outside > workspace – this is why it’s done separately!). I just have to fix the > Jenkins- > specific targets in our build.xml in Git repo to only do the actual > bootstrapping if Ivy is not installed locally on Developer’s/Jenkins’ machine. > Then it’s a no-op on most builds. For now the separate Jenkins job is enough > to quickly do the bootstrap as workaround. > > > > If nobody stops me from committing the following file into SVN: > > uschindler@lucene1-us-west:/x1/jenkins/lucene-data$ ls -lh > > total 2.9G > > -r--r--r-- 1 jenkins jenkins 2.9G May 9 2015 enwiki.random.lines.txt > > > > I will do this later this evening and change the Jenkins builds to do an > > extra > checkout into the workspace next to Git. > > I’d suggest to place the folder contents of ~/lucene-data here: > https://svn.apache.org/repos/asf/lucene/test-data > > > > Uwe > > > > ----- > > Uwe Schindler > > uschind...@apache.org > > ASF Member, Apache Lucene PMC / Committer > > Bremen, Germany > > http://lucene.apache.org/ > > > > From: Uwe Schindler [mailto:uschind...@apache.org] > > Sent: Monday, August 15, 2016 12:47 PM > > To: d...@lucene.apache.org; gmcdon...@apache.org; 'Uwe Schindler' > <uschind...@apache.org> > > Cc: 'builds' <builds@apache.org> > > Subject: RE: JDK, Maven, Ant versions have been consolidated. > > > > Hi, > > > > the test files could be a puppet module. > > > > The lucene.build.properties in the Jenkins home dir is a “hardware specific” > file, as it allows to configure the number of parallel runners and where the > special test files are – one would not be able to use it on any other jenkins > slave. If it is missing, defaults are used, which are not good for optimal CPU > use. We could add them also to the puppet module, but this would make it > harder for use to change the file easily. > > > > All other files are unmodified. > > > > Uwe > > > > ----- > > Uwe Schindler > > uschind...@apache.org > > ASF Member, Apache Lucene PMC / Committer > > Bremen, Germany > > http://lucene.apache.org/ > > > > From: Gav [mailto:gmcdon...@apache.org] > > Sent: Monday, August 15, 2016 5:36 AM > > To: Uwe Schindler <uschind...@apache.org> > > Cc: builds <builds@apache.org>; d...@lucene.apache.org > > Subject: Re: JDK, Maven, Ant versions have been consolidated. > > > > I'll make a copy of the jenkins home dir and get started. > > It might be an idea to add those test files to puppet wdyt? > > (I see nothing private about them) > > > > Gav... > > > > On Sat, Aug 13, 2016 at 3:13 AM, Uwe Schindler <uschind...@apache.org> > wrote: > >> Hi Gav, > >> > >> I disabled all auto-running Jenkins Jobs of Lucene/Solr as this spammed > the mailing list with like 50 mails today. Can you send me a msg when you > installed the required puppet modules? > >> > >> Uwe > >> > >> ----- > >> Uwe Schindler > >> uschind...@apache.org > >> ASF Member, Apache Lucene PMC / Committer > >> Bremen, Germany > >> http://lucene.apache.org/ > >> > >> From: Gav [mailto:gmcdon...@apache.org] > >> Sent: Friday, August 12, 2016 11:18 AM > >> To: Uwe Schindler <uschind...@apache.org> > >> Cc: builds <builds@apache.org> > >> Subject: Re: JDK, Maven, Ant versions have been consolidated. > >> > >> Hi Uwe, > >> > >> Will get on with your vm tomorrow. > >> > >> Gav... > >> > >> On Fri, Aug 12, 2016 at 6:39 PM, Uwe Schindler <uschind...@apache.org> > wrote: > >>> Hi, > >>> > >>> on the lucene-jenkins machine, the packages are still missing. I think you > have to install the jenkins puppet module. Here the message: > >>> FATAL: Cannot find executable from the chosen Ant installation "Ant > 1.8.4" > >>> > >>> See my e-mail from last week about using the "official" jenkins puppet > module (answer to your question). Unless our customizations in the home > dir keep preserved, we are fine with that: > >>> > >>> My mail: > >>> > >>> ==================================== > >>> Hi Gav, > >>> > >>> sorry for responding late – I am currently on vacation! > >>> > >>> About your questions: In the past we required some extra modules (like > older Subversion to be compatible with the older Jenkins Checkouts). As we > are on Git now and the build scripts use JGit to fetch revision numbers and > check for clean checkouts, this is obsolete. So if you want to install the > “Jenkins puppet module”, we are fine. > >>> > >>> There is one special case for Lucene: > >>> We have a hardware-specific “lucene.build.properties” file in Jenkin’s > root and also some additional “test files” (large wikiepdia checkouts) that > are > used for nightly tests. If the puppet module does not delete those files in > current ~/jenkins, it is ok. We would still need access to the machine and > sudo to jenkins. > >>> > >>> If this is all fine, go ahead! > >>> > >>> Uwe > >>> ======================================= > >>> > >>> Uwe > >>> > >>> ----- > >>> Uwe Schindler > >>> H.-H.-Meier-Allee 63, D-28213 Bremen > >>> http://www.thetaphi.de > >>> eMail: u...@thetaphi.de > >>> > >>> > -----Original Message----- > >>> > From: Gav [mailto:gmcdon...@apache.org] > >>> > Sent: Friday, August 12, 2016 6:43 AM > >>> > To: builds <builds@apache.org> > >>> > Subject: JDK, Maven, Ant versions have been consolidated. > >>> > > >>> > HI All, > >>> > > >>> > As per previous emails and as documented [1] [2] [3] we have now > >>> > consolidated the JDK, Maven and Ant install versions. > >>> > > >>> > All versions are now installed/managed via puppet and any 'auto- > install' > >>> > versions have been removed. > >>> > > >>> > At the time of consolidation and removal of versions, over 200 jobs > were > >>> > still configured to use deprecated versions despite the emails I've been > >>> > sending. Those 200+ jobs I have migrated myself to supported > versions. > >>> > > >>> > Any issues with your jobs please do create a new INFRA jira or email > this > >>> > list. > >>> > > >>> > Thanks, > >>> > > >>> > Gav... > >>> > > >>> > [1] - > >>> > > https://cwiki.apache.org/confluence/display/INFRA/JDK+Installation+Matrix > >>> > [2] - > >>> > > https://cwiki.apache.org/confluence/display/INFRA/Ant+Installation+Matrix > >>> > [3] - > >>> > > https://cwiki.apache.org/confluence/display/INFRA/Maven+Installation+Mat > >>> > rix > >>> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org
