+1 On Fri, May 27, 2011 at 7:48 PM, Jonathan Hsieh <j...@cloudera.com> wrote:
> Howdy! > > I would like to propose Flume to be an Apache Incubator project. Flume is > a > distributed, reliable, and available system for efficiently collecting, > aggregating, and moving large amounts of log data to scalable data storage > systems such as Apache Hadoop's HDFS. > > Here's a link to the proposal in the Incubator wiki > http://wiki.apache.org/incubator/FlumeProposal > > I've also pasted the initial contents below. > > Thanks! > Jon. > > = Flume - A Distributed Log Collection System = > > == Abstract == > > Flume is a distributed, reliable, and available system for efficiently > collecting, aggregating, and moving large amounts of log data to scalable > data storage systems such as Apache Hadoop's HDFS. > > == Proposal == > > Flume is a distributed, reliable, and available system for efficiently > collecting, aggregating, and moving large amounts of log data from many > different sources to a centralized data store. Its main goal is to deliver > data from applications to Hadoop’s HDFS. It has a simple and flexible > architecture for transporting streaming event data via flume nodes to the > data store. It is robust and fault-tolerant with tunable reliability > mechanisms that rely upon many failover and recovery mechanisms. The system > is centrally configured and allows for intelligent dynamic management. It > uses a simple extensible data model that allows for lightweight online > analytic applications. It provides a pluggable mechanism by which new > sources, destinations, and analytic functions which can be integrated > within > a Flume pipeline. > > == Background == > > Flume was initially developed by Cloudera to enable reliable and simplified > collection of log information from many distributed sources. It was later > open-sourced by Cloudera on GitHub as an Apache 2.0 licensed project in > June > 2010. During this time Flume has been formally released five times as > versions 0.9.0 (June 2010), 0.9.1 (Aug 2010), 0.9.1u1 (Oct 2010), 0.9.2 > (Nov > 2010), and 0.9.3 (Feb 2011). These releases are also distributed by > Cloudera as source and binaries along with enhancements as part of Cloudera > Distribution including Apache Hadoop (CDH). > > == Rationale == > > Collecting log information in a data center in a timely, reliable, and > efficient manner is a difficult challenge but important because when > aggregated and analyzed, log information can yield valuable business > insights. We believe that users and operators need a manageable > systematic > approach for log collection that simplifies the creation, the monitoring, > and the administration of reliable log data pipelines. Oftentimes today, > this collection is attempted by periodically shipping data in batches and > by > using potentially unreliable and inefficient ad-hoc methods. > > Log data is typically generated in various systems running within a data > center that can range from a few machines to hundreds of machines. In > aggregate, the data acts like a large-volume continuous stream with > contents > that can have highly-varied format and highly-varied content. The volume > and variety of raw log data makes Apache Hadoop's HDFS file system an ideal > storage location before the eventual analysis. Unfortunately, HDFS has > limitations with regards to durability as well as scaling limitations when > handling a large number of low-bandwidth connections or small files. > Similar technical challenges are also suffered when attempting to write > data to other data storage services. > > Flume addresses these challenges by providing a reliable, scalable, > manageable, and extensible solution. It uses a streaming design for > capturing and aggregating log information from varied sources in a > distributed environment and has centralized management features for minimal > configuration and management overhead. > > == Initial Goals == > > Flume is currently in its first major release with a considerable number of > enhancement requests, tasks, and issues recorded towards its future > development. The initial goal of this project will be to continue to build > community in the spirit of the "Apache Way", and to address the highly > requested features and bug-fixes towards the next dot release. > > Some goals include: > * To stand up a sustaining Apache-based community around the Flume > codebase. > * Implementing core functionality of a usable highly-available Flume > master. > * Performance, usability, and robustness improvements. > * Improving the ability to monitor and diagnose problems as data is > transported. > * Providing a centralized place for contributed connectors and related > projects. > > = Current Status = > > == Meritocracy == > > Flume was initially developed by Jonathan Hsieh in July 2009 along with > development team at Cloudera. Developers external to Cloudera provided > feedback, suggested features and fixes and implemented extensions of Flume. > Cloudera engineering team has since maintained the project with Jonathan > Hsieh, Henry Robinson, and Patrick Hunt dedicated towards its improvement. > Contributors to Flume and its connectors include developers from different > companies and different parts of the world. > > == Community == > > Flume is currently used by a number of organizations all over the world. > Flume has an active and growing user and developer community with active > participation in [user| > https://groups.google.com/a/cloudera.org/group/flume-user/topics] and > [developer|https://groups.google.com/a/cloudera.org/group/flume-dev/topics > ] > mailing lists. The users and developers also communicate via IRC on #flume > at irc.freenode.net. > > Since open sourcing the project, there have been over 15 different people > from diverse organizations who have contributed code. During this period, > the project team has hosted open, in-person, quarterly meetups to discuss > new features, new designs, and new use-case stories. > > == Core Developers == > > The core developers for Flume project are: > * Andrew Bayer: Andrew has a lot of expertise with build tools, > specifically Jenkins continuous integration and Maven. > * Jonathan Hsieh: Jonathan designed and implemented much of the original > code. > * Patrick Hunt: Patrick has improved the web interfaces of Flume > components > and contributed several build quality improvements. > * Bruce Mitchener: Bruce has improved the internal logging infrastructure > as well as edited significant portions of the Flume manual. > * Henry Robinson: Henry has implemented much of the ZooKeeper integration, > plugin mechanisms, as well as several Flume features and bug fixes. > * Eric Sammer: Eric has implemented the Maven build, as well as several > Flume features and bug fixes. > > All core developers of the Flume project have contributed towards Hadoop or > related Apache projects and are very familiar with Apache principals and > philosophy for community driven software development. > > == Alignment == > > Flume complements Hadoop Map-Reduce, Pig, Hive, HBase by providing a robust > mechanism to allow log data integration from external systems for effective > analysis. Its design enable efficient integration of newly ingested data > to > Hive's data warehouse. > > Flume's architecture is open and easily extensible. This has encouraged > many users to contribute integrate plugins to other projects. For example, > several users have contributed connectors to message queuing and bus > services, to several open source data stores, to incremental search > indexes, > and to a stream analysis engines. > > = Known Risks = > > == Orphaned Products == > > Flume is already deployed in production at multiple companies and they are > actively participating in feature requests and user led discussions. Flume > is getting traction with developers and thus the risks of it being orphaned > are minimal. > > == Inexperience with Open Source == > > All code developed for Flume has is open sourced by Cloudera under Apache > 2.0 license. All committers of Flume project are intimately familiar with > the Apache model for open-source development and are experienced with > working with new contributors. > > == Homogeneous Developers == > > The initial set of committers is from a reduced set of organizations. > However, we expect that once approved for incubation, the project will > attract new contributors from diverse organizations and will thus grow > organically. The participation of developers from several different > organizations in the mailing list is a strong indication for this > assertion. > > == Reliance on Salaried Developers == > > It is expected that Flume will be developed on salaried and volunteer time, > although all of the initial developers will work on it mainly on salaried > time. > > == Relationships with Other Apache Products == > > Flume depends upon other Apache Projects: Apache Hadoop, Apache Log4J, > Apache ZooKeeper, Apache Thrift, Apache Avro, multiple Apache Commons > components. Its build depends upon Apache Ant and Apache Maven. > > Flume users have created connectors that interact with several other Apache > projects including Apache HBase and Apache Cassandra. > > Flume's functionality has some indirect or direct overlap with the > functionality of Apache Chukwa but has several significant architectural > diffferences. Both systems can be used to collect log data to write to > hdfs. However, Chukwa's primary goals are the analytic and monitoring > aspects of a Hadoop cluster. Instead of focusing on analytics, Flume > focuses primarily upon data transport and integration with a wide set of > data sources and data destinations. Architecturally, Chukwa components > are > individually and statically configured. It also depends upon Hadoop > MapReduce for its core functionality. In contrast, Flume's components are > dynamically and centrally configured and does not depend directly upon > Hadoop MapReduce. Furthermore, Flume provides a more general model for > handling data and enables integration with projects such as Apache Hive, > data stores such as Apache HBase, Apache Cassandra and Voldemort, and > several Apache Lucene-related projects. > > == An Excessive Fascination with the Apache Brand == > > We would like Flume to become an Apache project to further foster a healthy > community of contributors and consumers around the project. Since Flume > directly interacts with many Apache Hadoop-related projects by solves an > important problem of many Hadoop users, residing in the the Apache Software > Foundation will increase interaction with the larger community. > > = Documentation = > > * All Flume documentation (User Guide, Developer Guide, Cookbook, and > Windows Guide) is maintained within Flume sources and can be built > directly. > * Cloudera provides documentation specific to its distribution of Flume > at: > http://archive.cloudera.com/cdh/3/flume/ > * Flume wiki at GitHub: https://github.com/cloudera/flume/wiki > * Flume jira at Cloudera: https://issues.cloudera.org/browse/flume > > = Initial Source = > > * https://github.com/cloudera/flume/tree/ > > == Source and Intellectual Property Submission Plan == > > * The initial source is already licensed under the Apache License, Version > 2.0. https://github.com/cloudera/flume/blob/master/LICENSE > > == External Dependencies == > > The required external dependencies are all Apache License or compatible > licenses. Following components with non-Apache licenses are enumerated: > > * org.arabidopsis.ahocorasick : BSD-style > > Non-Apache build tools that are used by Flume are as follows: > > * AsciiDoc: GNU GPLv2 > * FindBugs: GNU LGPL > * Cobertura: GNU GPLv2 > * PMD : BSD-style > > == Cryptography == > > Flume uses standard APIs and tools for SSH and SSL communication where > necessary. > > = Required Resources = > > == Mailing lists == > > * flume-private (with moderated subscriptions) > * flume-dev > * flume-commits > * flume-user > > == Subversion Directory == > > https://svn.apache.org/repos/asf/incubator/flume > > == Issue Tracking == > > JIRA Flume (FLUME) > > == Other Resources == > > The existing code already has unit and integration tests so we would like a > Hudson instance to run them whenever a new patch is submitted. This can be > added after project creation. > > = Initial Committers = > > * Andrew Bayer (abayer at cloudera dot com) > * Jonathan Hsieh (jon at cloudera dot com) > * Aaron Kimball (akimball83 at gmail dot com) > * Bruce Mitchener (bruce.mitchener at gmail dot com) > * Arvind Prabhakar (arvind at cloudera dot com) > * Ahmed Radwan (ahmed at cloudera dot com) > * Henry Robinson (henry at cloudera dot com) > * Eric Sammer (esammer at cloudera dot com) > > = Affiliations = > > * Andrew Bayer, Cloudera > * Jonathan Hsieh, Cloudera > * Aaron Kimball, Odiago > * Bruce Mitchener, Independent > * Arvind Prabhakar, Cloudera > * Ahmed Radwan, Cloudera > * Henry Robinson, Cloudera > * Eric Sammer, Cloudera > > > = Sponsors = > > == Champion == > > * Nigel Daley > > == Nominated Mentors == > > * Tom White > * Nigel Daley > > == Sponsoring Entity == > > * Apache Incubator PMC > > > -- > // Jonathan Hsieh (shay) > // Software Engineer, Cloudera > // j...@cloudera.com > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
