On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <sur...@hortonworks.com>wrote:
> With that we need to decide how our precommit process looks. > My inclination is to wait for +1 from precommit builds on > both the platforms to ensure no issues are introduced. > Thoughts? > > 2. Feature development impact > Some questions have been raised about would new features > need to be supported on both the platforms. Yes. I do not see a > reason why features cannot work on both the platforms, with > the exception of platform specific optimizations. This what Java > gives us. > > I'm concerned about the above. Personally, I don't have access to any Windows boxes with development tools, and I know nothing about developing on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated, for powerpoint :) If I submit a patch and it gets -1 "tests failed" on the Windows slave, how am I supposed to proceed? I think a reasonable compromise would be that the tests should always *build* on Windows before commit, and contributors should do their best to look at the test logs for any Windows-specific failures. But, beyond looking at the logs, a "-1 Tests failed on windows" should not block a commit. Those contributors who are interested in Windows being a first-class platform should be responsible for watching the Windows builds and debugging/fixing any regressions that might be Windows-specific. I also think the KDE model that Harsh pointed out is an interesting one -- ie the idea that we would not merge windows support to trunk, but rather treat is as a "parallel code line" which lives in the ASF and has its own builds and releases. The windows team would periodically merge trunk->win to pick up any new changes, and do a separate test/release process. I'm not convinced this is the best idea, but worth discussion of pros and cons. -Todd > > On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <e...@cloudera.com> wrote: > > > Bobby raises some good questions. A related one, since most current > > developers won't add Windows support for new features that are > > platform specific is it assumed that Windows development will either > > lag or will people actively work on keeping Windows up with the > > latest? And vice versa in case Windows support is implemented first. > > > > Is there a jira for resolving the outstanding TODOs in the code base > > (similar to HDFS-2148)? Looks like this merge doesn't introduce many > > which is great (just did a quick diff and grep). > > > > Thanks, > > Eli > > > > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <ev...@yahoo-inc.com> > wrote: > > > After this is merged in is Windows still going to be a second class > > > citizen but happens to work for more than just development or is it a > > > fully supported platform where if something breaks it can block a > > release? > > > How do we as a community intend to keep Windows support from breaking? > > > We don't have any Jenkins slaves to be able to run nightly tests to > > > validate everything still compiles/runs. This is not a blocker for me > > > because we often rely on individuals and groups to test Hadoop, but I > do > > > think we need to have this discussion before we put it in. > > > > > > --Bobby > > > > > > On 2/26/13 4:55 PM, "Suresh Srinivas" <sur...@hortonworks.com> wrote: > > > > > >>I had posted heads up about merging branch-trunk-win to trunk on Feb > 8th. > > >>I > > >>am happy to announce that we are ready for the merge. > > >> > > >>Here is a brief recap on the highlights of the work done: > > >>- Command-line scripts for the Hadoop surface area > > >>- Mapping the HDFS permissions model to Windows > > >>- Abstracted and reconciled mismatches around differences in Path > > >>semantics > > >>in Java and Windows > > >>- Native Task Controller for Windows > > >>- Implementation of a Block Placement Policy to support cloud > > >>environments, > > >>more specifically Azure. > > >>- Implementation of Hadoop native libraries for Windows (compression > > >>codecs, native I/O) > > >>- Several reliability issues, including race-conditions, intermittent > > test > > >>failures, resource leaks. > > >>- Several new unit test cases written for the above changes > > >> > > >>Please find the details of the work in CHANGES.branch-trunk-win.txt - > > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes< > http://bit.ly/13QOSo9 > > >, > > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the > work > > >>ported from branch-1-win to a branch based on trunk. > > >> > > >>For details of the testing done, please see the thread - > > >>http://bit.ly/WpavJ4. Merge patch for this is available on > HADOOP-8562< > > >>https://issues.apache.org/jira/browse/HADOOP-8562>. > > >> > > >>This was a large undertaking that involved developing code, testing the > > >>entire Hadoop stack, including scale tests. This is made possible only > > >>with > > >>the contribution from many many folks in the community. Following > people > > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas > Saha, > > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, > > Sumadhur > > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, > Thejas > > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya > > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, > > Suresh > > >>Srinivas and Sanjay Radia. There are many others who contributed as > well > > >>providing feedback and comments on numerous jiras. > > >> > > >>The vote will run for seven days and will end on March 5, 6:00PM PST. > > >> > > >>Regards, > > >>Suresh > > >> > > >> > > >> > > >> > > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman > > >><mah...@microsoft.com>wrote: > > >> > > >>> It is super exciting to look at the prospect of these changes being > > >>>merged > > >>> to trunk. Having Windows as one of the supported Hadoop platforms is > a > > >>> fantastic opportunity both for the Hadoop project and Microsoft > > >>>customers. > > >>> > > >>> This work began around a year back when a few of us started with a > > basic > > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft > > have > > >>> made significant progress in the following areas: > > >>> (PS: Some of these items are already included in Suresh's email, but > > >>> including again for completeness) > > >>> > > >>> - Command-line scripts for the Hadoop surface area > > >>> - Mapping the HDFS permissions model to Windows > > >>> - Abstracted and reconciled mismatches around differences in Path > > >>> semantics in Java and Windows > > >>> - Native Task Controller for Windows > > >>> - Implementation of a Block Placement Policy to support cloud > > >>> environments, more specifically Azure. > > >>> - Implementation of Hadoop native libraries for Windows (compression > > >>> codecs, native I/O) - Several reliability issues, including > > >>> race-conditions, intermittent test failures, resource leaks. > > >>> - Several new unit test cases written for the above changes > > >>> > > >>> In the process, we have closely engaged with the Apache open source > > >>> community and have got great support and assistance from the > community > > >>>in > > >>> terms of contributing fixes, code review comments and commits. > > >>> > > >>> In addition, the Hadoop team at Microsoft has also made good progress > > in > > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. > Many > > >>>of > > >>> these changes have already been committed to the respective trunks > with > > >>> help from various committers and contributors. It is great to see the > > >>> commitment of the community to support multiple platforms, and we > look > > >>> forward to the day when a developer/customer is able to successfully > > >>>deploy > > >>> a complete solution stack based on Apache Hadoop releases. > > >>> > > >>> Next Steps: > > >>> > > >>> All of the above changes are part of the Windows Azure HDInsight and > > >>> HDInsight Server products from Microsoft. We have successfully > > >>>on-boarded > > >>> several internal customers and have been running production workloads > > on > > >>> Windows Azure HDInsight. Our vision is to create a big data platform > > >>>based > > >>> on Hadoop, and we are committed to helping make Hadoop a world-class > > >>> solution that anyone can use to solve their biggest data challenges. > > >>> > > >>> As an immediate next step, we would like to have a discussion around > > how > > >>> we can ensure that the quality of the mainline Hadoop branches on > > >>>Windows > > >>> is maintained. To this end, we would like to get to the state where > we > > >>>have > > >>> pre-checkin validation gates and nightly test runs enabled on > Windows. > > >>>If > > >>> you have any suggestions around this, please do send an email. We > are > > >>> committed to helping sustain the long-term quality of Hadoop on both > > >>>Linux > > >>> and Windows. > > >>> > > >>> We sincerely thank the community for their contribution and support > so > > >>> far. And hope to continue having a close engagement in the future. > > >>> > > >>> -Microsoft HDInsight Team > > >>> > > >>> > > >>> -----Original Message----- > > >>> From: Suresh Srinivas [mailto:sur...@hortonworks.com] > > >>> Sent: Thursday, February 7, 2013 5:42 PM > > >>> To: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; > > >>> hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org > > >>> Subject: Heads up - merge branch-trunk-win to trunk > > >>> > > >>> The support for Hadoop on Windows was proposed in HADOOP-8079< > > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year > ago. > > >>>The > > >>> goal was to make Hadoop natively integrated, full-featured, and > > >>>performance > > >>> and scalability tuned on Windows Server or Windows Azure. > > >>> We are happy to announce that a lot of progress has been made in this > > >>> regard. > > >>> > > >>> Initial work started in a feature branch, branch-1-win, based on > > >>>branch-1. > > >>> The details related to the work done in the branch can be seen in > > >>> CHANGES.txt< > > >>> > > >>> > > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES > . > > >>>branch-1-win.txt?view=markup > > >>> >. > > >>> This work has been ported to a branch, branch-trunk-win, based on > > trunk. > > >>> Merge patch for this is available on > > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562> > > >>> . > > >>> > > >>> Highlights of the work done so far: > > >>> 1. Necessary changes in Hadoop to run natively on Windows. These > > changes > > >>> handle differences in platforms related to path names, process/task > > >>> management etc. > > >>> 2. Addition of winutils tools for managing file permissions and > > >>>ownership, > > >>> user group mapping, hardlinks, symbolic links, chmod, disk > utilization, > > >>>and > > >>> process/task management. > > >>> 3. Added cmd scripts equivalent to existing shell scripts > > >>> hadoop-daemon.sh, start and stop scripts. > > >>> 4. Addition of block placement policy implemnation to support cloud > > >>> enviroment, more specifically Azure. > > >>> > > >>> We are very close to wrapping up the work in branch-trunk-win and > > >>>getting > > >>> ready for a merge. Currently the merge patch is passing close to 100% > > of > > >>> unit tests on Linux. Soon I will call for a vote to merge this branch > > >>>into > > >>> trunk. > > >>> > > >>> Next steps: > > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work > > >>> completes and precommit build is clean. > > >>> 2. Start a discussion on adding Jenkins precommit builds on windows > and > > >>> how to integrate that with the existing commit process. > > >>> > > >>> Let me know if you have any questions. > > >>> > > >>> Regards, > > >>> Suresh > > >>> > > >>> > > >> > > >> > > >>-- > > >>http://hortonworks.com/download/ > > > > > > > > > -- > http://hortonworks.com/download/ > -- Todd Lipcon Software Engineer, Cloudera
