Hi Konstantin S, I accidentally merged some windows bug fixes to branch-2. However, branch-2 did not compile -- it showed that the windows changes were missing in branch-2. The ones causing compilation problems were already reverted (Thanks Sid and Suresh.) Sorry for the inconvenience.
Tsz-Wo ________________________________ From: Konstantin Shvachko <shv.had...@gmail.com> To: mapreduce-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; common-...@hadoop.apache.org; yarn-...@hadoop.apache.org Sent: Tuesday, March 26, 2013 5:25 AM Subject: Re: [Vote] Merge branch-trunk-win to trunk Andrew, this used to be on all -dev lists. Let's keep it that way. To the point. Does this mean that people are silently porting windows changes to branch-2? New features on a branch should be voted first, no? Thanks, --Konstantin On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <apurt...@apache.org> wrote: > Noticed this too. Simply a 'public' modifier is missing, but it's unclear > how this could not have been caught prior to check-in. > > > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <c...@apache.org> wrote: > >> It doesn't look like any progress has been done on the ticket below in the >> last 3 weeks. And now branch-2 can't be compiled because of >> >> >> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15] >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from >> outside package >> >> That's exactly why I was -1'ing this... >> Cos >> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote: >> > Thanks, gentlemen. I've opened and taken responsibility for >> > https://issues.apache.org/jira/browse/HADOOP-9359. Giri Kesavan has >> agreed >> > to help with the parts that require Jenkins admin access. >> > >> > Thanks, >> > --Matt >> > >> > >> > >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko < >> shv.had...@gmail.com>wrote: >> > >> > > +1 on the merge. >> > > >> > > I am glad we agreed. >> > > Having Jira to track the CI effort is a good idea. >> > > >> > > Thanks, >> > > --Konstantin >> > > >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfo...@hortonworks.com> >> wrote: >> > > > Thanks. I agree Windows -1's in test-patch should not block commits. >> > > > >> > > > --Matt >> > > > >> > > > >> > > > >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko < >> > > shv.had...@gmail.com> >> > > > wrote: >> > > >> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfo...@hortonworks.com >> > >> > > >> wrote: >> > > >> > Konstantine, you have voted -1, and stated some requirements >> before >> > > >> > you'll >> > > >> > withdraw that -1. As I plan to do work to fulfill those >> > > requirements, I >> > > >> > want to make sure that what I'm proposing will, in fact, satisfy >> you. >> > > >> > That's why I'm asking, if we implement full "test-patch" >> integration >> > > for >> > > >> > Windows, does it seem to you that that would provide adequate >> support? >> > > >> >> > > >> Yes. >> > > >> >> > > >> > I have learned not to presume that my interpretation is correct. >> My >> > > >> > interpretation of item #1 is that test-patch provides pre-commit >> > > build, >> > > >> > so >> > > >> > it would satisfy item #1. But rather than assuming that I am >> > > >> > interpreting >> > > >> > it correctly, I simply want your agreement that it would, or if >> not, >> > > >> > clarification why it won't. >> > > >> >> > > >> I agree it will satisfy my item #1. >> > > >> I did not agree in my previous email, but I changed my mind based on >> > > >> the latest discussion. I have to explain why now. >> > > >> I was proposing nightly build because I did not want pre-commit >> build >> > > >> for Windows block commits to Linux. But if people are fine just >> ignoring >> > > >> -1s for the Windows part of the build it should be good. >> > > >> >> > > >> > Regarding item #2, it is also my interpretation that test-patch >> > > provides >> > > >> > an >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit >> test, >> > > >> > with >> > > >> > logs available to the developer, so it would satisfy item #2. But >> > > >> > rather >> > > >> > than assuming that I am interpreting it correctly, I simply want >> your >> > > >> > agreement that it would, or if not, clarification why it won't. >> > > >> >> > > >> It will satisfy my item #2 in the following way: >> > > >> I can duplicate your pre-commit build for Windows and add an input >> > > >> parameter, which would let people run the build on their patches >> > > >> chosen from local machine rather than attaching them to Jiras. >> > > >> >> > > >> Thanks, >> > > >> --Konstantin >> > > >> >> > > >> > In agile terms, you are the Owner of these requirements. Please >> give >> > > me >> > > >> > owner feedback as to whether my proposed work sounds like it will >> > > >> > satisfy >> > > >> > the requirements. >> > > >> > >> > > >> > Thank you, >> > > >> > --Matt >> > > >> > >> > > >> > >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko >> > > >> > <shv.had...@gmail.com> >> > > >> > wrote: >> > > >> >> >> > > >> >> Didn't I explain in details what I am asking for? >> > > >> >> >> > > >> >> Thanks, >> > > >> >> --Konst >> > > >> >> >> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley < >> mfo...@hortonworks.com> >> > > >> >> wrote: >> > > >> >> > Hi Konstantin, >> > > >> >> > I'd like to point out two things: >> > > >> >> > First, I already committed in this thread (email of Thu, Feb >> 28, >> > > 2013 >> > > >> >> > at >> > > >> >> > 6:01 PM) to providing CI for Windows builds. So please stop >> acting >> > > >> >> > like >> > > >> >> > I'm >> > > >> >> > resisting this idea or something. >> > > >> >> > Second, you didn't answer my question, you just kvetched about >> the >> > > >> >> > phrasing. >> > > >> >> > So I ask again: >> > > >> >> > >> > > >> >> > Will providing full "test-patch" integration (pre-commit build >> and >> > > >> >> > unit >> > > >> >> > test >> > > >> >> > triggered by Jira "Patch Available" state) satisfy your >> request for >> > > >> >> > functionality #1 and #2? Yes or no, please. >> > > >> >> > >> > > >> >> > Thanks, >> > > >> >> > --Matt >> > > >> >> > >> > > >> >> > >> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko >> > > >> >> > <shv.had...@gmail.com> >> > > >> >> > wrote: >> > > >> >> >> >> > > >> >> >> Hi Matt, >> > > >> >> >> >> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley < >> > > mfo...@hortonworks.com> >> > > >> >> >> wrote: >> > > >> >> >> > Konstantin, >> > > >> >> >> > I would like to explore what it would take to remove this >> > > >> >> >> > perceived >> > > >> >> >> > impediment -- >> > > >> >> >> >> > > >> >> >> Glad you decided to explore. Thank you. >> > > >> >> >> >> > > >> >> >> > although I reserve the right to argue that this is not >> > > >> >> >> > pre-requisite to merging the cross-platform support patch. >> > > >> >> >> >> > > >> >> >> It's your right indeed. So as mine to question what the >> platform >> > > >> >> >> support means for you, which I believe remained unclear. >> > > >> >> >> I do not impede the change as you should have noticed. My >> > > >> >> >> requirement >> > > >> >> >> comes from my perception of the support, which means to me >> exactly >> > > >> >> >> two >> > > >> >> >> things: >> > > >> >> >> 1. The ability to recognise the code is broken for the >> platform >> > > >> >> >> 2. The ability to test new patches on the platform >> > > >> >> >> The latter is problematic, as many noticed in this thread, for >> > > those >> > > >> >> >> whose customary environment does not include Windows. >> > > >> >> >> >> > > >> >> >> > If we implemented full "test-patch" support for Windows on >> > > trunk, >> > > >> >> >> > would >> > > >> >> >> > that >> > > >> >> >> > fulfill both your items #1 and #2? Please note that: >> > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause >> a >> > > >> >> >> > pre-commit >> > > >> >> >> > build to start within, I believe, 20 minutes. >> > > >> >> >> > b) That build keeps logs for both java build and unit tests >> for >> > > >> >> >> > several >> > > >> >> >> > days, that are accessible to all viewers. >> > > >> >> >> >> > > >> >> >> In item #1 I mostly asking for the nightly build, which is >> simpler >> > > >> >> >> than "test-patch". The latter would be ideal from the platform >> > > >> >> >> support >> > > >> >> >> viewpoint, but it is for the community to decide if we want >> to add >> > > >> >> >> extra +3 hours to the build. >> > > >> >> >> Nightly build in my understanding is triggered by the timer >> rather >> > > >> >> >> than by Jira's "submit patch" button. On Jenkins build >> > > >> >> >> configuration >> > > >> >> >> you can specify it under "Build periodically". >> > > >> >> >> >> > > >> >> >> > So, does this provide sufficient on-demand support that we >> don't >> > > >> >> >> > have >> > > >> >> >> > to >> > > >> >> >> > implement a whole new on-demand VM support structure of some >> > > sort >> > > >> >> >> > for >> > > >> >> >> > #2 >> > > >> >> >> > (which would be an extraordinary and impractical demand)? >> > > >> >> >> >> > > >> >> >> I did not mention VMs. Item #2 means a build, which runs >> > > >> >> >> "test-patch" >> > > >> >> >> target with the file specified by a user (instead of a jira >> > > >> >> >> attachment). >> > > >> >> >> When user clicks "Build Now" link a box is displayed where the >> > > user >> > > >> >> >> can enter the file path containing the patch. This can be >> > > specified >> > > >> >> >> in >> > > >> >> >> the Build Configuration under "This build is parameterized" by >> > > >> >> >> choosing AddParameter / FileParameter. The build can run on >> the >> > > same >> > > >> >> >> Windows machine as the nightly build. >> > > >> >> >> Such build will let people test their patches for Windows on >> > > Jenkins >> > > >> >> >> if they don't posses a license for the right version of >> Windows. >> > > >> >> >> I hope this will not turn into extraordinary or impractical >> > > effort. >> > > >> >> >> >> > > >> >> >> Thanks, >> > > >> >> >> --Konst >> > > >> >> >> >> > > >> >> >> > Thanks, >> > > >> >> >> > --Matt >> > > >> >> >> > >> > > >> >> >> > >> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko >> > > >> >> >> > <shv.had...@gmail.com> >> > > >> >> >> > wrote: >> > > >> >> >> >> >> > > >> >> >> >> -1 >> > > >> >> >> >> We should have a CI infrastructure in place before we can >> > > commit >> > > >> >> >> >> to >> > > >> >> >> >> supporting Windows platform. >> > > >> >> >> >> >> > > >> >> >> >> Eric is right Win/Cygwin was supported since day one. >> > > >> >> >> >> I had a Windows box under my desk running nightly builds >> back >> > > in >> > > >> >> >> >> 2006-07. >> > > >> >> >> >> People were irritated but I was filing windows bugs until >> 0.22 >> > > >> >> >> >> release. >> > > >> >> >> >> Times changing and I am glad to see wider support for Win >> > > >> >> >> >> platform. >> > > >> >> >> >> >> > > >> >> >> >> But in order to make it work you guys need to put the CI >> > > process >> > > >> >> >> >> in >> > > >> >> >> >> place >> > > >> >> >> >> >> > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit. >> > > >> >> >> >> - Nightly would mean that changes can be committed to trunk >> > > based >> > > >> >> >> >> on >> > > >> >> >> >> linux PreCommit build. And people will file bugs if the >> change >> > > >> >> >> >> broke >> > > >> >> >> >> Windows nightly build. >> > > >> >> >> >> - PreCommit-win build will mean automatic reporting failed >> > > tests >> > > >> >> >> >> to >> > > >> >> >> >> respective jira blocking commits the same way as it is now >> with >> > > >> >> >> >> linux >> > > >> >> >> >> PreCommit builds. >> > > >> >> >> >> We should discuss which way is more efficient for >> developers. >> > > >> >> >> >> >> > > >> >> >> >> 2. On-demand-windows Jenkins build. >> > > >> >> >> >> I see it as a build to which I can attach my patch and the >> > > build >> > > >> >> >> >> will >> > > >> >> >> >> run my changes on a dedicated windows box. >> > > >> >> >> >> That way people can test their changes without having >> personal >> > > >> >> >> >> windows >> > > >> >> >> >> nodes. >> > > >> >> >> >> >> > > >> >> >> >> I think this is the minimal set of requirement for us to be >> > > able >> > > >> >> >> >> to >> > > >> >> >> >> commit to the new platform. >> > > >> >> >> >> Right now I see only one windows related build >> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/ >> > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the >> > > last >> > > >> >> >> >> month. >> > > >> >> >> >> >> > > >> >> >> >> Thanks, >> > > >> >> >> >> --Konst >> > > >> >> >> >> >> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler >> > > >> >> >> >> <eri...@hortonworks.com> wrote: >> > > >> >> >> >> > +1 (non-binding) >> > > >> >> >> >> > >> > > >> >> >> >> > A few of observations: >> > > >> >> >> >> > >> > > >> >> >> >> > - Windows has actually been a supported platform for >> Hadoop >> > > >> >> >> >> > since >> > > >> >> >> >> > 0.1 >> > > >> >> >> >> > . >> > > >> >> >> >> > Doug championed supporting windows then and we've >> continued >> > > to >> > > >> >> >> >> > do >> > > >> >> >> >> > it >> > > >> >> >> >> > with >> > > >> >> >> >> > varying vigor over time. To my knowledge we've never >> made a >> > > >> >> >> >> > decision >> > > >> >> >> >> > to >> > > >> >> >> >> > drop windows support. The change here is improving our >> > > support >> > > >> >> >> >> > and >> > > >> >> >> >> > dropping >> > > >> >> >> >> > the requirement of cigwin. We had Nutch windows users >> on the >> > > >> >> >> >> > list >> > > >> >> >> >> > in >> > > >> >> >> >> > 2006 >> > > >> >> >> >> > and we've been supporting windows FS requirements since >> > > >> >> >> >> > inception. >> > > >> >> >> >> > >> > > >> >> >> >> > - A little pragmatism will go a long way. As a community >> > > we've >> > > >> >> >> >> > got >> > > >> >> >> >> > to >> > > >> >> >> >> > stay committed to keeping hadoop simple (so it does work >> on >> > > >> >> >> >> > many >> > > >> >> >> >> > platforms) >> > > >> >> >> >> > and extending it to take advantage of key emerging >> > > OS/hardware >> > > >> >> >> >> > features, >> > > >> >> >> >> > such as containers, new FSs, virtualization, flash ... >> We >> > > >> >> >> >> > should >> > > >> >> >> >> > all >> > > >> >> >> >> > plan >> > > >> >> >> >> > to let new features & optimizations emerge that don't >> work >> > > >> >> >> >> > everywhere, if >> > > >> >> >> >> > they are compelling and central to hadoop's mission of >> being >> > > >> >> >> >> > THE >> > > >> >> >> >> > best >> > > >> >> >> >> > fabric >> > > >> >> >> >> > for storing and processing big data. >> > > >> >> >> >> > >> > > >> >> >> >> > - A UI project like KDE has to deal with the MANY >> differences >> > > >> >> >> >> > between >> > > >> >> >> >> > windows and linux UI APIs. Hadoop faces no such complex >> > > >> >> >> >> > challenge >> > > >> >> >> >> > and hence >> > > >> >> >> >> > can be maintained from a single codeline IMO. It is >> mostly >> > > >> >> >> >> > abstracted from >> > > >> >> >> >> > the OS APIs via Java and our design choices. Where it >> is not >> > > >> >> >> >> > we >> > > >> >> >> >> > can >> > > >> >> >> >> > continue to add plugable abstractions. >> > > >> >> >> >> > >> > > >> >> >> > >> > > >> >> >> > >> > > >> >> > >> > > >> >> > >> > > >> > >> > > >> > >> > > > >> > > > >> > > >> > > > > -- > Best regards, > > - Andy > > Problems worthy of attack prove their worth by hitting back. - Piet Hein > (via Tom White)
