The collisions have been happening for about a year now. The frequency is increasing, but not enough to be particularly worrisome. (So I'm slightly amused that one blowing up is suddenly a major freakout.)
Making changes to the configuration without knowing what one is doing is probably a bad idea. For example, if people are removing the shared cache, I hope they're also prepared for the bitching that is going to go with the extremely significant slow down caused by downloading the java prereqs for building for every test... As far as Yetus goes, we've got a JIRA open to provide for per-instance caches when using the docker container code. I've got it in my head how I think we can do it, but just haven't had a chance to code it. So once that gets written up + turning on containers should make the problem go away without any significant impact on test time. Of course, that won't help the scheduled builds but those happen at an even smaller rate. On Sep 18, 2015, at 12:19 PM, Andrew Wang <andrew.w...@cloudera.com> wrote: > Sangjin, you should have access to the precommit jobs if you log in with > your Apache credentials, even as a branch committer. > > https://builds.apache.org/job/PreCommit-HDFS-Build/configure > > The actual maven invocation is managed by test-patch.sh though. > test-patch.sh has a MAVEN_ARGS which looks like what we want, but I don't > think we can just set it before calling test-patch, since it'd get squashed > by setup_defaults. > > Allen/Chris/Yetus folks, any guidance here? > > Thanks, > Andrew > > On Fri, Sep 18, 2015 at 11:55 AM, <e...@zusammenkunft.net> wrote: > >> You can use one per build processor, that reduces concurrent updates but >> still keeps the cache function. And then try to avoid using install. >> >> -- >> http://bernd.eckenfels.net >> >> -----Original Message----- >> From: Andrew Wang <andrew.w...@cloudera.com> >> To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org> >> Cc: Andrew Bayer <andrew.ba...@gmail.com>, Sangjin Lee <sj...@twitter.com>, >> Lei Xu <l...@cloudera.com>, infrastruct...@apache.org >> Sent: Fr., 18 Sep. 2015 20:42 >> Subject: Re: Local repo sharing for maven builds >> >> I think each job should use a maven.repo.local within its workspace like >> abayer said. This means lots of downloading, but it's isolated. >> >> If we care about download time, we could also bootstrap with a tarred >> .m2/repository after we've run a `mvn compile`, so before it installs the >> hadoop artifacts. >> >> On Fri, Sep 18, 2015 at 11:02 AM, Ming Ma <min...@twitter.com.invalid> >> wrote: >> >>> +hadoop common dev. Any suggestions? >>> >>> >>> On Fri, Sep 18, 2015 at 10:41 AM, Andrew Bayer <andrew.ba...@gmail.com> >>> wrote: >>> >>>> You can change your maven call to use a different repository - I >> believe >>>> you do that with -Dmaven.repository.local=path/to/repo >>>> On Sep 18, 2015 19:39, "Ming Ma" <min...@twitter.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> We are seeing some strange behaviors in HDFS precommit build. It seems >>>>> like it is caused by the local repo on the same machine being used by >>>>> different concurrent jobs which can cause issues. >>>>> >>>>> In HDFS, the build and test of "hadoop-hdfs-project/hdfs" depend on >>>>> "hadoop-hdfs-project/hdfs-client"'s hadoop-hdfs-client-3.0.0- >>>>> SNAPSHOT.jar. HDFS-9004 adds some new method to >>> hadoop-hdfs-client-3.0.0-SNAPSHOT.jar. >>>>> In the precommit build for HDFS-9004, unit tests for >>> "hadoop-hdfs-project/hdfs" >>>>> complain the method isn't defined >>>>> https://builds.apache.org/job/PreCommit-HDFS-Build/12522/testReport/. >>>>> Interestingly sometimes it just works fine >>>>> https://builds.apache.org/job/PreCommit-HDFS-Build/12507/testReport/. >>>>> >>>>> So we are suspecting that there is another job running at the same >> time >>>>> that published different version of >>> hadoop-hdfs-client-3.0.0-SNAPSHOT.jar >>>>> which doesn't have the new methods defined to the local repo which is >>>>> shared by all jobs on that machine. >>>>> >>>>> If the above analysis is correct, what is the best way to fix the >> issue >>>>> so that different jobs can use their own maven local repo for build >> and >>>>> test? >>>>> >>>>> Thanks. >>>>> >>>>> Ming >>>>> >>>> >>> >>
