Let me add a bit more about the feasibility of this. I have been doing some experiments by duplicating some common code code in hdfs-only , and yarn/MR only; and am able to build and use hdfs independently.
Now that bigtop has matured, we can still do a single distro in apache with independently released mr/yarn and hdfs. That will enable parallel development, and will also reduce stabilization overload at a mega-release time. If HDFS is released independently, with its own RPC and protocol versions, features such as pluggable namespaces will not have to wait for the next mega-release of the entire stack. Would love to hear what hdfs developers think about this. - milind Sent from my iPhone > On Oct 10, 2013, at 20:31, Milind Bhandarkar <mbhandar...@gopivotal.com> > wrote: > > ( this message is not intended for specific folks, by mistake, but for all > the hdfs-dev list, deliberately;) > > Hello Folks, > > I do not want to scratch the already bleeding wounds, and want to resolve > these issues amicably, without causing a big inter-vendor confrontation. > > So, these are the facts, as I (and several others in the hadoop community) > see this. > > 1. there was an attempt to separate different hadoop projects, such as > common, hdfs, mapreduce. > > 2. that attempt was aborted because of several things. common ownership, i.e. > committership being the biggest issue. > > 3. in the meanwhile, several important, release-worthy, hdfs improvements > were committed to Hadoop. (Thats why I supported Konst's appeal for 0.22. And > also incorporated into Hadoop products by the largest hadoop ecosystem > contributor, and several others.) > > 4. All the apache hadoop bylaws were followed, to get these improvements into > Hadoop project. > > 5. Yet, common project, which is not even a top-level project, since the > awkward re-merge happened, got an invompatible wire-protocol change, which > was accepted and promoted by a specific section, in spite of kicking and > screaming of (what I think of) a representative of a large hadoop user > community. > > 6. That, and such other changes, has created a big issue for a part of the > community which has tested hdfs part of 2.x and has spent a lot of efforts to > stabilize hdfs, since this was the major part of assault from proprietary > storage systems, such as You-Know-Who. > > I would like to raise this issue as an individual, regardless of my > affiliation, so that, we can make hdfs worthy of its association with the top > level ecosystem, without being closely associated with it. > > What do the hdfs developers think? > > - milind > > Sent from my iPhone
