Cool project and nice proposal :-) Tommaso
2011/5/29 Nigel Daley <nda...@mac.com> > +1 on the proposal. Looking forward to the vote. > > Nige > > On May 28, 2011, at 10:49 PM, Mattmann, Chris A (388J) wrote: > > > On May 27, 2011, at 11:40 AM, arv...@cloudera.com wrote: > > > >> Greetings All, > >> > >> We would like to propose Sqoop Project for inclusion in ASF Incubator as > a > >> new podling. Sqoop is a tool designed for efficiently transferring bulk > data > >> between Apache Hadoop and structured datastores such as relational > >> databases. The complete proposal can be found at: > >> > >> http://wiki.apache.org/incubator/SqoopProposal > >> > >> The initial contents of this proposal are also pasted below for > convenience. > >> > >> Thanks and Regards, > >> Arvind Prabhakar > >> > >> = Sqoop - A Data Transfer Tool for Hadoop = > >> > >> == Abstract == > >> > >> Sqoop is a tool designed for efficiently transferring bulk data between > >> Apache Hadoop and structured datastores such as relational databases. > You > >> can use Sqoop to import data from external structured datastores into > Hadoop > >> Distributed File System or related systems like Hive and HBase. > Conversely, > >> Sqoop can be used to extract data from Hadoop and export it to external > >> structured datastores such as relational databases and enterprise data > >> warehouses. > >> > >> == Proposal == > >> > >> Hadoop and related systems operate on large volumes of data. Typically > this > >> data originates from outside of Hadoop infrastructure and must be > >> provisioned for consumption by Hadoop and related systems for analysis > and > >> processing. Sqoop allows fast provisioning of data into Hadoop and > related > >> systems by providing a bulk import and export mechanism that enables > >> consumers to effectively use Hadoop for data analysis and processing. > >> > >> == Background == > >> > >> Sqoop was initially developed by Cloudera to enable the import and > export of > >> data between various databases and Hadoop Distributed File System > (HDFS). It > >> was provided as a patch to Hadoop project via the issue [[ > >> https://issues.apache.org/jira/browse/HADOOP-5815|HADOOP-5815]] and was > >> maintained as a contrib module to Hadoop between May 2009 to April 2010. > In > >> April 2010, Sqoop was removed from Hadoop contrib via [[ > >> https://issues.apache.org/jira/browse/MAPREDUCE-1644|MAPREDUCE-1644]] > and > >> was made available by Cloudera on [[ > http://github.com/cloudera/sqoop|GitHub]]. > >> > >> > >> Since then Sqoop has been maintained by Cloudera as an open source > project > >> on GitHub. All code available in Sqoop is open source and made publicaly > >> available under the Apache 2 license. During this time Sqoop has been > >> formally released three times as versions 1.0, 1.1 and 1.2. > >> > >> == Rationale == > >> > >> Hadoop is often used to process data that originated or is later served > by > >> structured data stores such as relational databases, spreadsheets or > >> enterprise data warehouses. Unfortunately, current methods of > transferring > >> data are inefficient and ad hoc, often consisting of manual steps > specific > >> to the external system. These steps are necessary to help provision this > >> data for consumption by Map-Reduce jobs, or by systems that build on top > of > >> Hadoop such as Hive and Pig. The transfer of this data can take > substantial > >> amount of time depending upon its size. An optimal transfer approach > that > >> works well with one particular datastore will typically not work as > >> optimally with another datastore due to inherent architectural > differences > >> between different datastore implementations. Sqoop addresses this > problem by > >> providing connectivity of Hadoop with external systems via pluggable > >> connectors. Specialized connectors are developed for optimal performance > for > >> data transfer between Hadoop and target systems. > >> > >> Analyzed and processed data from Hadoop and related systems may also > require > >> to be provisioned outside of Hadoop for consumption by business > >> applications. Sqoop allows the export of data from Hadoop to external > >> systems to facilitate its use in other systems. This too, like the > import > >> scenario, is implemented via specialized connectors that are built for > the > >> purposes of optimal integration between Hadoop and external systems. > >> > >> Connectors can be built for systems that Sqoop does not yet integrate > with > >> and thus can be easily incorporated into Sqoop. Connectors allow Sqoop > to > >> interface with external systems of different types, ensuring that newer > >> systems can integrate with Hadoop with relative ease and in a consistent > >> manner. > >> > >> Besides allowing integration with other external systems, Sqoop provides > >> tight integration with systems that build on to of Hadoop such as Hive, > >> HBase etc - thus providing data integration between Hadoop based systems > and > >> external systems in a single step manner. > >> > >> == Initial Goals == > >> > >> Sqoop is currently in its first major release with a considerable number > of > >> enhancement requests, tasks, and issues logged towards its future > >> development. The initial goal of this project will be to address the > highly > >> requested features and bug-fixes towards its next dot release. The key > >> features of interest are the following: > >> * Support for bulk import into Apache HBase. > >> * Allow user to supply password in permission protected file. > >> * Support for pluggable query to help Sqoop identify the metadata > >> associated with the source or target table definitions. > >> * Allow user to specify custom split semantics for efficient > >> parallelization of import jobs. > >> > >> = Current Status = > >> > >> == Meritocracy == > >> > >> Sqoop has been an open source project since its start. It was initially > >> developed by Aaron Kimball in May 2009 along with development team at > >> Cloudera and supplied as a patch to Hadoop project. Later it was moved > to > >> GitHub as a Cloudera open-source project where Cloudera engineering team > has > >> since maintained it with Arvind Prabhakar and Ahmed Radwan dedicated > towards > >> its improvement. Developers external to Cloudera provided feedback, > >> suggested features and fixes and implemented extensions of Sqoop since > its > >> inception. Contributors to Sqoop include developers from different > >> companies and different parts of the world. > >> > >> == Community == > >> > >> Sqoop is currently used by a number of organizations all over the world. > >> Sqoop has an active and growing user community with active participation > in > >> [[https://groups.google.com/a/cloudera.org/group/sqoop-user/topics|user > ]] > >> and [[ > >> > https://groups.google.com/a/cloudera.org/group/sqoop-dev/topics|developer > ]] > >> mailing lists. > >> > >> == Core Developers == > >> > >> The core developers for Sqoop project are: > >> * Aaron Kimball: Aaron designed and implemented much of the original > code. > >> * Arvind Prabhakar: Has been working on Sqoop features and bug fixes. > >> * Ahmed Radwan: Has been working on Sqoop features and bug fixes. > >> * Jonathan Hsieh: Has started working on Sqoop features and bug fixes. > >> * Other contributors to the project include: Angus He, Brian Muller, Eli > >> Collins, Guy Le Mar, James Grant, Konstantin Boudnik, Lars Francke, > Michael > >> Hausler, Michael Katzenellenbogen, Pter Happ and Scott Foster. > >> > >> All committers to Sqoop project have contributed towards Hadoop or > related > >> Apache projects and are very familiar with Apache principals and > philosophy > >> for community driven software development. > >> > >> == Alignment == > >> > >> Sqoop complements Hadoop Map-Reduce, Pig, Hive, HBase by providing a > robust > >> mechanism to allow data integration from external systems for effective > data > >> analysis. It integrates with Hive and HBase currently and work is being > done > >> to integrate it with Pig. > >> > >> = Known Risks = > >> > >> == Orphaned Products == > >> > >> Sqoop is already deployed in production at multiple companies and they > are > >> actively participating in feature requests and user led discussions. > Sqoop > >> is getting traction with developers and thus the risks of it being > orphaned > >> are minimal. > >> > >> == Inexperience with Open Source == > >> > >> All code developed for Sqoop has been open source from the start. The > >> initial part of Sqoop development was done within Hadoop project as a > >> contrib module. Since then it has been maintained as an Apache 2.0 > licensed > >> open-source project on GitHub by Cloudera. > >> > >> All committers of Sqoop project are intimately familiar with the Apache > >> model for open-source development and are experienced with working with > new > >> contributors. Aaron Kimball, the creator of the project and one of the > >> committers is also a committer on Apache MapReduce. > >> > >> == Homogeneous Developers == > >> > >> The initial set of committers is from a small set of organizations. > However, > >> we expect that once approved for incubation, the project will attract > new > >> contributors from diverse organizations and will thus grow organically. > The > >> participation of developers from several different organizations in the > >> mailing list is a strong indication for this assertion. > >> > >> == Reliance on Salaried Developers == > >> > >> It is expected that Sqoop will be developed on salaried and volunteer > time, > >> although all of the initial developers will work on it mainly on > salaried > >> time. > >> > >> == Relationships with Other Apache Products == > >> > >> Sqoop depends upon other Apache Projects: Hadoop, Hive, HBase Log4J and > >> multiple Apache commons components and build systems like Ant and Maven. > >> > >> == An Excessive Fascination with the Apache Brand == > >> > >> The reasons for joining Apache are to increase the synergy with other > Apache > >> Hadoop related projects and to foster a healthy community of > contributors > >> and consumers around the project. This is facilitated by ASF and that is > the > >> primary reason we would like Sqoop to become an Apache project. > >> > >> = Documentation = > >> > >> * All Sqoop documentation is maintained within Sqoop sources and can be > >> built directly. > >> * Sqoop docs: http://archive.cloudera.com/cdh/3/sqoop/ > >> * Sqoop wiki at GitHub: https://github.com/cloudera/sqoop/wiki > >> * Sqoop jira at Cloudera: https://issues.cloudera.org/browse/sqoop > >> > >> = Initial Source = > >> > >> * https://github.com/cloudera/sqoop/tree/ > >> > >> == Source and Intellectual Property Submission Plan == > >> > >> * The initial source is already Apache 2.0 licensed. > >> > >> == External Dependencies == > >> > >> The required external dependencies are all Apache License or compatible > >> licenses. Following components with non-Apache licenses are enumerated: > >> > >> * HSQLDB: HSQLDB License - a BSD-based license. > >> > >> Non-Apache build tools that are used by Sqoop are as follows: > >> > >> * AsciiDoc: GNU GPLv2 > >> * Checkstyle: GNU LGPLv3 > >> * FindBugs: GNU LGPL > >> * Cobertura: GNU GPLv2 > >> > >> == Cryptography == > >> > >> Sqoop does not depend upon any cryptography tools or libraries. > >> > >> = Required Resources = > >> > >> == Mailing lists == > >> > >> * sqoop-private (with moderated subscriptions) > >> * sqoop-dev > >> * sqoop-commits > >> * sqoop-user > >> > >> == Subversion Directory == > >> > >> https://svn.apache.org/repos/asf/incubator/sqoop > >> > >> == Issue Tracing == > >> > >> JIRA Sqoop (SQOOP) > >> > >> == Other Resources == > >> > >> The existing code already has unit and integration tests so we would > like a > >> Hudson instance to run them whenever a new patch is submitted. This can > be > >> added after project creation. > >> > >> = Initial Committers = > >> > >> * Arvind Prabhakar (arvind at cloudera dot com) > >> * Ahmed Radwan (a dot aboelela at gmail dot com) > >> * Jonathan Hsieh (jon at cloudera dot com) > >> * Aaron Kimball (kimballa at apache dot org) > >> * Greg Cottman (greg dot cottman at quest dot com) > >> * Guy le Mar (guy dot lemar at quest dot com) > >> * Roman Shaposhnik (rvs at cloudera dot com) > >> * Andrew Bayer (andrew at cloudera dot com) > >> > >> A CLA is already on file for Aaron Kimball. > >> > >> = Affiliations = > >> > >> * Arvind Prabhakar, Cloudera > >> * Ahmed Radwan, Cloudera > >> * Jonathan Hsieh, Cloudera > >> * Aaron Kimball, Odiago > >> * Greg Cottman, Quest > >> * Guy le Mar, Quest > >> * Roman Shaposhnik, Cloudera > >> * Andrew Bayer, Cloudera > >> > >> = Sponsors = > >> > >> == Champion == > >> > >> * Tom White (tomwhite at apache dot org) > >> > >> == Nominated Mentors == > >> > >> * Patrick Hunt (phunt at apache dot org) > >> > >> == Sponsoring Entity == > >> > >> * Apache Incubator PMC > > > > > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Chris Mattmann, Ph.D. > > Senior Computer Scientist > > NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA > > Office: 171-266B, Mailstop: 171-246 > > Email: chris.a.mattm...@nasa.gov > > WWW: http://sunset.usc.edu/~mattmann/ > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > Adjunct Assistant Professor, Computer Science Department > > University of Southern California, Los Angeles, CA 90089 USA > > ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > > For additional commands, e-mail: general-h...@incubator.apache.org > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
