Thanks Uwe for moving this process along. +1 to make a new SVN dir at https://svn.apache.org/repos/asf/lucene/test-data/ and put the enwiki data file there.
-- Steve www.lucidworks.com > On Aug 15, 2016, at 1:47 PM, Uwe Schindler <[email protected]> wrote: > > Hi, > > I thought about the wikipedia test files used by Jenkins: I think I would > commit them to SVN inside the Lucene/Solr project’s folder. The Nightly-Test > Jenkins jobs that use them could simply check them out into a separate dir of > the workspace. Byt that it is easier for us to update them. If you also think > this is fine, we can leave it like that. > > The ~jenkins/lucene.build.properties file is (as said before) > machine/hardware specific. I’d keep them in Jenkins’ home dir. We may create > a puppet module out of it, but this would prevent us from optimizing or > changing them quickly per project requirements. > > Finally the “ant ivy-bootstrap” task was for now separated into a Jenkins > job, that can be manually run > (https://builds.apache.org/job/Lucene-Ivy-Bootstrap/). In the future, we can > add the “ivy-bootstrap” target as dependency to the “ant jenkins” and other > targets which are solely triggered by Jenkins, so separately bootstrapping is > no longer required (please keep in mind that bootstrapping places the ivy.jar > file in ~/.ant/lib, so outside workspace – this is why it’s done > separately!). I just have to fix the Jenkins-specific targets in our > build.xml in Git repo to only do the actual bootstrapping if Ivy is not > installed locally on Developer’s/Jenkins’ machine. Then it’s a no-op on most > builds. For now the separate Jenkins job is enough to quickly do the > bootstrap as workaround. > > If nobody stops me from committing the following file into SVN: > uschindler@lucene1-us-west:/x1/jenkins/lucene-data$ ls -lh > total 2.9G > -r--r--r-- 1 jenkins jenkins 2.9G May 9 2015 enwiki.random.lines.txt > > I will do this later this evening and change the Jenkins builds to do an > extra checkout into the workspace next to Git. > I’d suggest to place the folder contents of ~/lucene-data here: > https://svn.apache.org/repos/asf/lucene/test-data > > Uwe > > ----- > Uwe Schindler > [email protected] > ASF Member, Apache Lucene PMC / Committer > Bremen, Germany > http://lucene.apache.org/ > > From: Uwe Schindler [mailto:[email protected]] > Sent: Monday, August 15, 2016 12:47 PM > To: [email protected]; [email protected]; 'Uwe Schindler' > <[email protected]> > Cc: 'builds' <[email protected]> > Subject: RE: JDK, Maven, Ant versions have been consolidated. > > Hi, > > the test files could be a puppet module. > > The lucene.build.properties in the Jenkins home dir is a “hardware specific” > file, as it allows to configure the number of parallel runners and where the > special test files are – one would not be able to use it on any other jenkins > slave. If it is missing, defaults are used, which are not good for optimal > CPU use. We could add them also to the puppet module, but this would make it > harder for use to change the file easily. > > All other files are unmodified. > > Uwe > > ----- > Uwe Schindler > [email protected] > ASF Member, Apache Lucene PMC / Committer > Bremen, Germany > http://lucene.apache.org/ > > From: Gav [mailto:[email protected]] > Sent: Monday, August 15, 2016 5:36 AM > To: Uwe Schindler <[email protected]> > Cc: builds <[email protected]>; [email protected] > Subject: Re: JDK, Maven, Ant versions have been consolidated. > > I'll make a copy of the jenkins home dir and get started. > It might be an idea to add those test files to puppet wdyt? > (I see nothing private about them) > > Gav... > > On Sat, Aug 13, 2016 at 3:13 AM, Uwe Schindler <[email protected]> wrote: >> Hi Gav, >> >> I disabled all auto-running Jenkins Jobs of Lucene/Solr as this spammed the >> mailing list with like 50 mails today. Can you send me a msg when you >> installed the required puppet modules? >> >> Uwe >> >> ----- >> Uwe Schindler >> [email protected] >> ASF Member, Apache Lucene PMC / Committer >> Bremen, Germany >> http://lucene.apache.org/ >> >> From: Gav [mailto:[email protected]] >> Sent: Friday, August 12, 2016 11:18 AM >> To: Uwe Schindler <[email protected]> >> Cc: builds <[email protected]> >> Subject: Re: JDK, Maven, Ant versions have been consolidated. >> >> Hi Uwe, >> >> Will get on with your vm tomorrow. >> >> Gav... >> >> On Fri, Aug 12, 2016 at 6:39 PM, Uwe Schindler <[email protected]> wrote: >>> Hi, >>> >>> on the lucene-jenkins machine, the packages are still missing. I think you >>> have to install the jenkins puppet module. Here the message: >>> FATAL: Cannot find executable from the chosen Ant installation "Ant 1.8.4" >>> >>> See my e-mail from last week about using the "official" jenkins puppet >>> module (answer to your question). Unless our customizations in the home dir >>> keep preserved, we are fine with that: >>> >>> My mail: >>> >>> ==================================== >>> Hi Gav, >>> >>> sorry for responding late – I am currently on vacation! >>> >>> About your questions: In the past we required some extra modules (like >>> older Subversion to be compatible with the older Jenkins Checkouts). As we >>> are on Git now and the build scripts use JGit to fetch revision numbers and >>> check for clean checkouts, this is obsolete. So if you want to install the >>> “Jenkins puppet module”, we are fine. >>> >>> There is one special case for Lucene: >>> We have a hardware-specific “lucene.build.properties” file in Jenkin’s root >>> and also some additional “test files” (large wikiepdia checkouts) that are >>> used for nightly tests. If the puppet module does not delete those files in >>> current ~/jenkins, it is ok. We would still need access to the machine and >>> sudo to jenkins. >>> >>> If this is all fine, go ahead! >>> >>> Uwe >>> ======================================= >>> >>> Uwe >>> >>> ----- >>> Uwe Schindler >>> H.-H.-Meier-Allee 63, D-28213 Bremen >>> http://www.thetaphi.de >>> eMail: [email protected] >>> >>> > -----Original Message----- >>> > From: Gav [mailto:[email protected]] >>> > Sent: Friday, August 12, 2016 6:43 AM >>> > To: builds <[email protected]> >>> > Subject: JDK, Maven, Ant versions have been consolidated. >>> > >>> > HI All, >>> > >>> > As per previous emails and as documented [1] [2] [3] we have now >>> > consolidated the JDK, Maven and Ant install versions. >>> > >>> > All versions are now installed/managed via puppet and any 'auto-install' >>> > versions have been removed. >>> > >>> > At the time of consolidation and removal of versions, over 200 jobs were >>> > still configured to use deprecated versions despite the emails I've been >>> > sending. Those 200+ jobs I have migrated myself to supported versions. >>> > >>> > Any issues with your jobs please do create a new INFRA jira or email this >>> > list. >>> > >>> > Thanks, >>> > >>> > Gav... >>> > >>> > [1] - >>> > https://cwiki.apache.org/confluence/display/INFRA/JDK+Installation+Matrix >>> > [2] - >>> > https://cwiki.apache.org/confluence/display/INFRA/Ant+Installation+Matrix >>> > [3] - >>> > https://cwiki.apache.org/confluence/display/INFRA/Maven+Installation+Mat >>> > rix >>> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
