Here's a daily build for hadoop-2 branch https://builds.apache.org/job/Hadoop-branch2
It just builds the full Hadoop project without running any tests (for now). Can be easily extended to do test runs/artifact deployment, if needed. Cos On Mon, Mar 25, 2013 at 07:14PM, Konstantin Boudnik wrote: > Yes, you are right of course - the mis-merged commit is the cause. Thanks for > pointing this out! > > I think it would be beneficial if we had branch-2 on going build in the > Jenkins. I will go ahead and create one tonight. > > Cos > > On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote: > > Adding other mailing lists I missed earlier. > > > > Cos, > > > > There is progress being made on that ticket. Also it has nothing to do with > > that. > > > > Please follow the discussion here and why this happened due to an invalid > > commit that was reverted - > > https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650 > > > > Regards, > > Suresh > > > > > > On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <c...@apache.org> wrote: > > > > > It doesn't look like any progress has been done on the ticket below in the > > > last 3 weeks. And now branch-2 can't be compiled because of > > > > > > > > > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] > > > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed > > > from > > > outside package > > > > > > That's exactly why I was -1'ing this... > > > Cos > > > > > > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: > > > > Thanks, gentlemen. I've opened and taken responsibility for > > > > https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has > > > agreed > > > > to help with the parts that require Jenkins admin access. > > > > > > > > Thanks, > > > > --Matt > > > > > > > > > > > > > > > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko < > > > shv.had...@gmail.com>wrote: > > > > > > > > > +1 on the merge. > > > > > > > > > > I am glad we agreed. > > > > > Having Jira to track the CI effort is a good idea. > > > > > > > > > > Thanks, > > > > > --Konstantin > > > > > > > > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfo...@hortonworks.com> > > > wrote: > > > > > > Thanks. I agree Windows -1's in test-patch should not block > > > > > > commits. > > > > > > > > > > > > --Matt > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko < > > > > > shv.had...@gmail.com> > > > > > > wrote: > > > > > >> > > > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfo...@hortonworks.com > > > > > > > > > >> wrote: > > > > > >> > Konstantine, you have voted -1, and stated some requirements > > > before > > > > > >> > you'll > > > > > >> > withdraw that -1. As I plan to do work to fulfill those > > > > > requirements, I > > > > > >> > want to make sure that what I'm proposing will, in fact, satisfy > > > you. > > > > > >> > That's why I'm asking, if we implement full "test-patch" > > > integration > > > > > for > > > > > >> > Windows, does it seem to you that that would provide adequate > > > support? > > > > > >> > > > > > >> Yes. > > > > > >> > > > > > >> > I have learned not to presume that my interpretation is correct. > > > My > > > > > >> > interpretation of item #1 is that test-patch provides pre-commit > > > > > build, > > > > > >> > so > > > > > >> > it would satisfy item #1. But rather than assuming that I am > > > > > >> > interpreting > > > > > >> > it correctly, I simply want your agreement that it would, or if > > > not, > > > > > >> > clarification why it won't. > > > > > >> > > > > > >> I agree it will satisfy my item #1. > > > > > >> I did not agree in my previous email, but I changed my mind based > > > > > >> on > > > > > >> the latest discussion. I have to explain why now. > > > > > >> I was proposing nightly build because I did not want pre-commit > > > build > > > > > >> for Windows block commits to Linux. But if people are fine just > > > ignoring > > > > > >> -1s for the Windows part of the build it should be good. > > > > > >> > > > > > >> > Regarding item #2, it is also my interpretation that test-patch > > > > > provides > > > > > >> > an > > > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit > > > test, > > > > > >> > with > > > > > >> > logs available to the developer, so it would satisfy item #2. > > > > > >> > But > > > > > >> > rather > > > > > >> > than assuming that I am interpreting it correctly, I simply want > > > your > > > > > >> > agreement that it would, or if not, clarification why it won't. > > > > > >> > > > > > >> It will satisfy my item #2 in the following way: > > > > > >> I can duplicate your pre-commit build for Windows and add an input > > > > > >> parameter, which would let people run the build on their patches > > > > > >> chosen from local machine rather than attaching them to Jiras. > > > > > >> > > > > > >> Thanks, > > > > > >> --Konstantin > > > > > >> > > > > > >> > In agile terms, you are the Owner of these requirements. Please > > > give > > > > > me > > > > > >> > owner feedback as to whether my proposed work sounds like it will > > > > > >> > satisfy > > > > > >> > the requirements. > > > > > >> > > > > > > >> > Thank you, > > > > > >> > --Matt > > > > > >> > > > > > > >> > > > > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko > > > > > >> > <shv.had...@gmail.com> > > > > > >> > wrote: > > > > > >> >> > > > > > >> >> Didn't I explain in details what I am asking for? > > > > > >> >> > > > > > >> >> Thanks, > > > > > >> >> --Konst > > > > > >> >> > > > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley < > > > mfo...@hortonworks.com> > > > > > >> >> wrote: > > > > > >> >> > Hi Konstantin, > > > > > >> >> > I'd like to point out two things: > > > > > >> >> > First, I already committed in this thread (email of Thu, Feb > > > 28, > > > > > 2013 > > > > > >> >> > at > > > > > >> >> > 6:01 PM) to providing CI for Windows builds. So please stop > > > acting > > > > > >> >> > like > > > > > >> >> > I'm > > > > > >> >> > resisting this idea or something. > > > > > >> >> > Second, you didn't answer my question, you just kvetched about > > > the > > > > > >> >> > phrasing. > > > > > >> >> > So I ask again: > > > > > >> >> > > > > > > >> >> > Will providing full "test-patch" integration (pre-commit build > > > and > > > > > >> >> > unit > > > > > >> >> > test > > > > > >> >> > triggered by Jira "Patch Available" state) satisfy your > > > request for > > > > > >> >> > functionality #1 and #2? Yes or no, please. > > > > > >> >> > > > > > > >> >> > Thanks, > > > > > >> >> > --Matt > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko > > > > > >> >> > <shv.had...@gmail.com> > > > > > >> >> > wrote: > > > > > >> >> >> > > > > > >> >> >> Hi Matt, > > > > > >> >> >> > > > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley < > > > > > mfo...@hortonworks.com> > > > > > >> >> >> wrote: > > > > > >> >> >> > Konstantin, > > > > > >> >> >> > I would like to explore what it would take to remove this > > > > > >> >> >> > perceived > > > > > >> >> >> > impediment -- > > > > > >> >> >> > > > > > >> >> >> Glad you decided to explore. Thank you. > > > > > >> >> >> > > > > > >> >> >> > although I reserve the right to argue that this is not > > > > > >> >> >> > pre-requisite to merging the cross-platform support patch. > > > > > >> >> >> > > > > > >> >> >> It's your right indeed. So as mine to question what the > > > platform > > > > > >> >> >> support means for you, which I believe remained unclear. > > > > > >> >> >> I do not impede the change as you should have noticed. My > > > > > >> >> >> requirement > > > > > >> >> >> comes from my perception of the support, which means to me > > > exactly > > > > > >> >> >> two > > > > > >> >> >> things: > > > > > >> >> >> 1. The ability to recognise the code is broken for the > > > platform > > > > > >> >> >> 2. The ability to test new patches on the platform > > > > > >> >> >> The latter is problematic, as many noticed in this thread, > > > > > >> >> >> for > > > > > those > > > > > >> >> >> whose customary environment does not include Windows. > > > > > >> >> >> > > > > > >> >> >> > If we implemented full "test-patch" support for Windows on > > > > > trunk, > > > > > >> >> >> > would > > > > > >> >> >> > that > > > > > >> >> >> > fulfill both your items #1 and #2? Please note that: > > > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause > > > a > > > > > >> >> >> > pre-commit > > > > > >> >> >> > build to start within, I believe, 20 minutes. > > > > > >> >> >> > b) That build keeps logs for both java build and unit tests > > > for > > > > > >> >> >> > several > > > > > >> >> >> > days, that are accessible to all viewers. > > > > > >> >> >> > > > > > >> >> >> In item #1 I mostly asking for the nightly build, which is > > > simpler > > > > > >> >> >> than "test-patch". The latter would be ideal from the > > > > > >> >> >> platform > > > > > >> >> >> support > > > > > >> >> >> viewpoint, but it is for the community to decide if we want > > > to add > > > > > >> >> >> extra +3 hours to the build. > > > > > >> >> >> Nightly build in my understanding is triggered by the timer > > > rather > > > > > >> >> >> than by Jira's "submit patch" button. On Jenkins build > > > > > >> >> >> configuration > > > > > >> >> >> you can specify it under "Build periodically". > > > > > >> >> >> > > > > > >> >> >> > So, does this provide sufficient on-demand support that we > > > don't > > > > > >> >> >> > have > > > > > >> >> >> > to > > > > > >> >> >> > implement a whole new on-demand VM support structure of > > > > > >> >> >> > some > > > > > sort > > > > > >> >> >> > for > > > > > >> >> >> > #2 > > > > > >> >> >> > (which would be an extraordinary and impractical demand)? > > > > > >> >> >> > > > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs > > > > > >> >> >> "test-patch" > > > > > >> >> >> target with the file specified by a user (instead of a jira > > > > > >> >> >> attachment). > > > > > >> >> >> When user clicks "Build Now" link a box is displayed where > > > > > >> >> >> the > > > > > user > > > > > >> >> >> can enter the file path containing the patch. This can be > > > > > specified > > > > > >> >> >> in > > > > > >> >> >> the Build Configuration under "This build is parameterized" > > > > > >> >> >> by > > > > > >> >> >> choosing AddParameter / FileParameter. The build can run on > > > the > > > > > same > > > > > >> >> >> Windows machine as the nightly build. > > > > > >> >> >> Such build will let people test their patches for Windows on > > > > > Jenkins > > > > > >> >> >> if they don't posses a license for the right version of > > > Windows. > > > > > >> >> >> I hope this will not turn into extraordinary or impractical > > > > > effort. > > > > > >> >> >> > > > > > >> >> >> Thanks, > > > > > >> >> >> --Konst > > > > > >> >> >> > > > > > >> >> >> > Thanks, > > > > > >> >> >> > --Matt > > > > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko > > > > > >> >> >> > <shv.had...@gmail.com> > > > > > >> >> >> > wrote: > > > > > >> >> >> >> > > > > > >> >> >> >> -1 > > > > > >> >> >> >> We should have a CI infrastructure in place before we can > > > > > commit > > > > > >> >> >> >> to > > > > > >> >> >> >> supporting Windows platform. > > > > > >> >> >> >> > > > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one. > > > > > >> >> >> >> I had a Windows box under my desk running nightly builds > > > back > > > > > in > > > > > >> >> >> >> 2006-07. > > > > > >> >> >> >> People were irritated but I was filing windows bugs until > > > 0.22 > > > > > >> >> >> >> release. > > > > > >> >> >> >> Times changing and I am glad to see wider support for Win > > > > > >> >> >> >> platform. > > > > > >> >> >> >> > > > > > >> >> >> >> But in order to make it work you guys need to put the CI > > > > > process > > > > > >> >> >> >> in > > > > > >> >> >> >> place > > > > > >> >> >> >> > > > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit. > > > > > >> >> >> >> - Nightly would mean that changes can be committed to > > > > > >> >> >> >> trunk > > > > > based > > > > > >> >> >> >> on > > > > > >> >> >> >> linux PreCommit build. And people will file bugs if the > > > change > > > > > >> >> >> >> broke > > > > > >> >> >> >> Windows nightly build. > > > > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed > > > > > tests > > > > > >> >> >> >> to > > > > > >> >> >> >> respective jira blocking commits the same way as it is now > > > with > > > > > >> >> >> >> linux > > > > > >> >> >> >> PreCommit builds. > > > > > >> >> >> >> We should discuss which way is more efficient for > > > developers. > > > > > >> >> >> >> > > > > > >> >> >> >> 2. On-demand-windows Jenkins build. > > > > > >> >> >> >> I see it as a build to which I can attach my patch and the > > > > > build > > > > > >> >> >> >> will > > > > > >> >> >> >> run my changes on a dedicated windows box. > > > > > >> >> >> >> That way people can test their changes without having > > > personal > > > > > >> >> >> >> windows > > > > > >> >> >> >> nodes. > > > > > >> >> >> >> > > > > > >> >> >> >> I think this is the minimal set of requirement for us to > > > > > >> >> >> >> be > > > > > able > > > > > >> >> >> >> to > > > > > >> >> >> >> commit to the new platform. > > > > > >> >> >> >> Right now I see only one windows related build > > > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/ > > > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in > > > > > >> >> >> >> the > > > > > last > > > > > >> >> >> >> month. > > > > > >> >> >> >> > > > > > >> >> >> >> Thanks, > > > > > >> >> >> >> --Konst > > > > > >> >> >> >> > > > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler > > > > > >> >> >> >> <eri...@hortonworks.com> wrote: > > > > > >> >> >> >> > +1 (non-binding) > > > > > >> >> >> >> > > > > > > >> >> >> >> > A few of observations: > > > > > >> >> >> >> > > > > > > >> >> >> >> > - Windows has actually been a supported platform for > > > Hadoop > > > > > >> >> >> >> > since > > > > > >> >> >> >> > 0.1 > > > > > >> >> >> >> > . > > > > > >> >> >> >> > Doug championed supporting windows then and we've > > > continued > > > > > to > > > > > >> >> >> >> > do > > > > > >> >> >> >> > it > > > > > >> >> >> >> > with > > > > > >> >> >> >> > varying vigor over time. To my knowledge we've never > > > made a > > > > > >> >> >> >> > decision > > > > > >> >> >> >> > to > > > > > >> >> >> >> > drop windows support. The change here is improving our > > > > > support > > > > > >> >> >> >> > and > > > > > >> >> >> >> > dropping > > > > > >> >> >> >> > the requirement of cigwin. We had Nutch windows users > > > on the > > > > > >> >> >> >> > list > > > > > >> >> >> >> > in > > > > > >> >> >> >> > 2006 > > > > > >> >> >> >> > and we've been supporting windows FS requirements since > > > > > >> >> >> >> > inception. > > > > > >> >> >> >> > > > > > > >> >> >> >> > - A little pragmatism will go a long way. As a > > > > > >> >> >> >> > community > > > > > we've > > > > > >> >> >> >> > got > > > > > >> >> >> >> > to > > > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work > > > on > > > > > >> >> >> >> > many > > > > > >> >> >> >> > platforms) > > > > > >> >> >> >> > and extending it to take advantage of key emerging > > > > > OS/hardware > > > > > >> >> >> >> > features, > > > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ... > > > We > > > > > >> >> >> >> > should > > > > > >> >> >> >> > all > > > > > >> >> >> >> > plan > > > > > >> >> >> >> > to let new features & optimizations emerge that don't > > > work > > > > > >> >> >> >> > everywhere, if > > > > > >> >> >> >> > they are compelling and central to hadoop's mission of > > > being > > > > > >> >> >> >> > THE > > > > > >> >> >> >> > best > > > > > >> >> >> >> > fabric > > > > > >> >> >> >> > for storing and processing big data. > > > > > >> >> >> >> > > > > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY > > > differences > > > > > >> >> >> >> > between > > > > > >> >> >> >> > windows and linux UI APIs. Hadoop faces no such complex > > > > > >> >> >> >> > challenge > > > > > >> >> >> >> > and hence > > > > > >> >> >> >> > can be maintained from a single codeline IMO. It is > > > mostly > > > > > >> >> >> >> > abstracted from > > > > > >> >> >> >> > the OS APIs via Java and our design choices. Where it > > > is not > > > > > >> >> >> >> > we > > > > > >> >> >> >> > can > > > > > >> >> >> >> > continue to add plugable abstractions. > > > > > >> >> >> >> > > > > > > >> >> >> > > > > > > >> >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > http://hortonworks.com/download/
signature.asc
Description: Digital signature
