I see a couple blockers that prevent this from being a contrib project
of Hadoop:
1) The Hadoop project is actively trying to remove the contrib
projects it has, see http://tinyurl.com/6yl25jz. I doubt it's
interested in any new ones.
2) The Hadoop project is producing a release every 2 or 3 years
currently. As a new project Howl will be wanting to release every 2
or 3 months for a while. Being tied to something as slow moving as
Hadoop for releases would make it hard for Howl get releases out the
door.
Alan.
On Feb 2, 2011, at 4:57 PM, Milind Bhandarkar wrote:
I feel that Howl should start as a contrib to Hadoop, and move to be
a subproject of Hadoop once there is sufficient adoption, rather
than going the incubator way. My reasons are as follows:
1. Howl is aimed at providing abstractions for facilitating
interoperability between various systems built *on top of Hadoop*,
and should not limit itself to Pig, Hive, and native MapReduce. So,
any system that is hadoop compatible should be able to use Howl as a
metadata store.
2. Having Howl as contrib of Hadoop will ensure that the input and
output formats, compression codecs, underlying storage APIs etc
remain in sync from release to release, and users do not have to
worry about whether version x of Howl is compatible with version y
of Hadoop or not.
3. Pig, Hive, Cascading, .. are all already dependent on Hadoop.
Including Howl as Hadoop contrib means they do not add any more
dependencies.
4. The roadmap of Howl includes authentication and authorization
support. It is a standard industry practice that metadata security
mechanisms match those for data security. Thus, a significant code
can be shared with hadoop's authorization and authentication.
5. Hadoop-compatible file systems provide an abstraction over
underlying storage systems. Howl currently provides a table
abstraction over the file system. In future, when Hadoop provides
blockpool abstraction (as part of federation), Howl will be able to
take advantage of that and optimize.
6. Howl roadmap currently does not contain multi-tenancy features
such as quotas. Since there is a strong correlation between number
of tables, number of partitions in Howl and number of directories
and files in HDFS, it could be streamlined if Howl is part of Hadoop.
Thoughts ?
- milind
On Feb 2, 2011, at 1:18 PM, Alan Gates wrote:
Howl is a table management system built to provide metadata and
storage management across data processing tools in Hadoop (Pig,
Hive, MapReduce, ...). You can learn more details at http://wiki.apache.org/pig/Howl
. For the last six months the code has been hosted at github. The
Howl team would like to move the project into the Apache
Incubator. You can see the proposal for the project at http://wiki.apache.org/incubator/HowlProposal
.
In order to be accepted as an Incubator project Howl needs a
Sponsoring project. I propose that we, the Pig project, sponsor
Howl. By sponsoring Howl we are saying that we believe it is a
good fit for the ASF and that we will assist the Howl project to
succeed. You can read full details of sponsoring a project at http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Sponsor
.
Our bylaws don't explicitly cover such a vote, but I think lazy
majority should be reasonable. All votes are welcome, PMC member
votes will be binding.
Clearly I'm +1.
Alan.
---
Milind Bhandarkar
[email protected]
