I feel that Howl should start as a contrib to Hadoop, and move to be a subproject of Hadoop once there is sufficient adoption, rather than going the incubator way. My reasons are as follows:
1. Howl is aimed at providing abstractions for facilitating interoperability between various systems built *on top of Hadoop*, and should not limit itself to Pig, Hive, and native MapReduce. So, any system that is hadoop compatible should be able to use Howl as a metadata store. 2. Having Howl as contrib of Hadoop will ensure that the input and output formats, compression codecs, underlying storage APIs etc remain in sync from release to release, and users do not have to worry about whether version x of Howl is compatible with version y of Hadoop or not. 3. Pig, Hive, Cascading, .. are all already dependent on Hadoop. Including Howl as Hadoop contrib means they do not add any more dependencies. 4. The roadmap of Howl includes authentication and authorization support. It is a standard industry practice that metadata security mechanisms match those for data security. Thus, a significant code can be shared with hadoop's authorization and authentication. 5. Hadoop-compatible file systems provide an abstraction over underlying storage systems. Howl currently provides a table abstraction over the file system. In future, when Hadoop provides blockpool abstraction (as part of federation), Howl will be able to take advantage of that and optimize. 6. Howl roadmap currently does not contain multi-tenancy features such as quotas. Since there is a strong correlation between number of tables, number of partitions in Howl and number of directories and files in HDFS, it could be streamlined if Howl is part of Hadoop. Thoughts ? - milind On Feb 2, 2011, at 1:18 PM, Alan Gates wrote: > Howl is a table management system built to provide metadata and storage > management across data processing tools in Hadoop (Pig, Hive, MapReduce, > ...). You can learn more details at http://wiki.apache.org/pig/Howl. For > the last six months the code has been hosted at github. The Howl team would > like to move the project into the Apache Incubator. You can see the proposal > for the project at http://wiki.apache.org/incubator/HowlProposal. > > In order to be accepted as an Incubator project Howl needs a Sponsoring > project. I propose that we, the Pig project, sponsor Howl. By sponsoring > Howl we are saying that we believe it is a good fit for the ASF and that we > will assist the Howl project to succeed. You can read full details of > sponsoring a project at > http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Sponsor. > > Our bylaws don't explicitly cover such a vote, but I think lazy majority > should be reasonable. All votes are welcome, PMC member votes will be > binding. > > Clearly I'm +1. > > Alan. --- Milind Bhandarkar [email protected]
