+1 from us also - at neustar are working on a large deployment using kafka as well. We'd be interested to help in the future.
-jeff On Fri, Jun 24, 2011 at 2:17 PM, Henry Saputra <henry.sapu...@gmail.com>wrote: > +1 > > A very good proposal and it seems to help solve our need for low > latency event messaging system, so looking forward to it. > > I would love to contribute to the project and have added my name to > list initial committers if no objection. > > - Henry > > >> 2011/6/22 Jun Rao <jun...@gmail.com> > >> > >> > Hi, > >> > > >> > I would like to propose Kafka to be an Apache Incubator project. > Kafka > >> is > >> > a > >> > distributed, high throughput, publish-subscribe system for processing > >> large > >> > amounts of streaming data. > >> > > >> > Here's a link to the proposal in the Incubator wiki > >> > http://wiki.apache.org/incubator/KafkaProposal > >> > > >> > I've also pasted the initial contents below. > >> > > >> > Thanks, > >> > > >> > Jun > >> > > >> > == Abstract == > >> > Kafka is a distributed publish-subscribe system for processing large > >> > amounts > >> > of streaming data. > >> > > >> > == Proposal == > >> > Kafka provides an extremely high throughput distributed > publish/subscribe > >> > messaging system. Additionally, it supports relatively long term > >> > persistence of messages to support a wide variety of consumers, > >> > partitioning > >> > of the message stream across servers and consumers, and functionality > for > >> > loading data into Apache Hadoop for offline, batch processing. > >> > > >> > == Background == > >> > Kafka was developed at LinkedIn to process the large amounts of events > >> > generated by that company's website and provide a common repository > for > >> > many > >> > types of consumers to access and process those events. Kafka has been > >> used > >> > in production at LinkedIn scale to handle dozens of types of events > >> > including page views, searches and social network activity. Kafka > >> clusters > >> > at LinkedIn currently process more than two billion events per day. > >> > > >> > Kafka fills the gap between messaging systems such as Apache ActiveMQ, > >> > which > >> > can provide high-volume messaging systems but lack persistence of > those > >> > messages, and log processing systems such as Scribe and Flume, which > do > >> not > >> > provide adequate latency for our diverse set of consumers. Kafka can > >> also > >> > be inserted into traditional log-processing systems, acting as an > >> > intermediate step before further processing. Kafka focuses > relentlessly > >> on > >> > performance and throughput by not introspecting into message content, > nor > >> > indexing them on the broker. We also achieve high performance by > >> depending > >> > on Java's sendFile/transferTo capabilities to minimize intermediate > >> buffer > >> > copies and relying on the OS's pagecache to efficiently serve up > message > >> > contents to consumers. > >> > > >> > Kafka is written in Scala and depends on Apache ZooKeeper for > >> coordination > >> > amongst its producers, brokers and consumers. > >> > > >> > Kafka was developed internally at LinkedIn to meet our particular use > >> > cases, > >> > but will be useful to many organizations facing a similar need to > >> reliably > >> > process large amounts of streaming data. Therefore, we would like to > >> share > >> > it the ASF and begin developing a community of developers and users > >> within > >> > Apache. > >> > > >> > == Rationale == > >> > Many organizations can benefit from a reliable stream processing > system > >> > such > >> > as Kafka. While our use case of processing events from a very large > >> > website > >> > like LinkedIn has driven the design of Kafka, its uses are varied and > we > >> > expect many new use cases to emerge. Kafka provides a natural bridge > >> > between near real-time event processing and offline batch processing > and > >> > will appeal to many users. > >> > > >> > == Current Status == > >> > === Meritocracy === > >> > Our intent with this incubator proposal is to start building a diverse > >> > developer community around Kafka following the Apache meritocracy > model. > >> > Since Kafka was open sourced we have solicited contributions via the > >> > website > >> > and presentations given to user groups and technical audiences. We > have > >> > had > >> > positive responses to these and have received several contributions > and > >> > clients for other languages. We plan to continue this support for new > >> > contributors and work with those who contribute significantly to the > >> > project > >> > to make them committers. > >> > > >> > === Community === > >> > Kafka is currently being used by developed by engineers within > LinkedIn > >> and > >> > used in production in that company. Additionally, we have active users > in > >> > or > >> > have received contributions from a diverse set of companies including > >> > MediaSift, SocialTwist, Clearspring and Urban Airship. Recent public > >> > presentations of Kafka and its goals garnered much interest from > >> potential > >> > contributors. We hope to extend our contributor base significantly and > >> > invite all those who are interested in building high-throughput > >> distributed > >> > systems to participate. We have begun receiving contributions from > >> outside > >> > of LinkedIn, including clients for several languages including Ruby, > PHP, > >> > Clojure, .NET and Python. > >> > > >> > To further this goal, we use GitHub issue tracking and branching > >> > facilities, > >> > as well as maintaining a public mailing list via Google Groups. > >> > > >> > === Core Developers === > >> > Kafka is currently being developed by four engineers at LinkedIn: Neha > >> > Narkhede, Jun Rao, Jakob Homan and Jay Kreps. Jun has experience > within > >> > Apache as a Cassandra committer and PMC member. Neha has been an > active > >> > contributor to several projects LinkedIn has open sourced, including > >> Bobo, > >> > Sensei and Zoie. Jay has experience with open source software as the > >> > originator of the Project Voldemort project, as well as being active > >> within > >> > the Hadoop ecosystem community. Jakob is an Apache Hadoop committer > and > >> PMC > >> > and previous Apache ZooKeeper contributor. > >> > > >> > === Alignment === > >> > The ASF is the natural choice to host the Kafka project as its goal of > >> > encouraging community-driven open-source projects fits with our vision > >> for > >> > Kafka. Additionally, many other projects with which we are familiar > with > >> > and expect Kafka to integrate with, such as Apache Hadoop, Pig, > ZooKeeper > >> > and log4j are hosted by the ASF and we will benefit and provide > benefit > >> by > >> > close proximity to them. > >> > > >> > == Known Risks == > >> > === Orphaned Products === > >> > The core developers plan to work full time on the project. There is > very > >> > little risk of Kafka being abandoned as it is a critical part of > >> LinkedIn's > >> > internal infrastructure and is in production use. > >> > > >> > === Inexperience with Open Source === > >> > All of the core developers have experience with open source > development. > >> > LinkedIn open sourced Kafka several months ago and has been receiving > >> > contributions since. Jun is an Apache Cassandra committer and PMC > >> member. > >> > Jay and Neha have been involved with several open source projects > >> released > >> > by LinkedIn. Jakob has been actively involved with the ASF as a > >> full-time > >> > Hadoop committer and PMC member. > >> > > >> > === Homogeneous Developers === > >> > The current core developers are all from LinkedIn. However, we hope to > >> > establish a developer community that includes contributors from > several > >> > corporations and we actively encouraging new contributors via the > mailing > >> > lists and public presentations of Kafka. > >> > > >> > === Reliance on Salaried Developers === > >> > Currently, the developers are paid to do work on Kafka. However, once > the > >> > project has a community built around it, we expect to get committers, > >> > developers and community from outside the current core developers. > >> However, > >> > because LinkedIn relies on Kafka internally, the reliance on salaried > >> > developers is unlikely to change. > >> > > >> > === Relationships with Other Apache Products === > >> > Kafka is deeply integrated with Apache products. Kafka uses Apache > >> > ZooKeeper > >> > to coordinate its state amongst the brokers, consumers, and soon, the > >> > producers. Kafka provides input formats to allow Hadoop MapReduce to > >> load > >> > data directly from Kafka. Kafka provides an appender to allow > consuming > >> > data directly from Apache log4j. > >> > > >> > === An Excessive Fascination with the Apache Brand === > >> > While we respect the reputation of the Apache brand and have no doubts > >> that > >> > it will attract contributors and users, our interest is primarily to > give > >> > Kafka a solid home as an open source project following an established > >> > development model. We have also given reasons in the Rationale and > >> > Alignment > >> > sections. > >> > > >> > == Documentation == > >> > Information about Kafka can be found at [ > http://sna-projects.com/kafka/] > >> > The > >> > following links provide more information about the project: > >> > > >> > * Kafka roadmap and goals: [ > http://sna-projects.com/kafka/projects.php] > >> > * The GitHub site: [https://github.com/kafka-dev/kafka] > >> > * Kafka overview from Jay Kreps: [ > >> > http://www.slideshare.net/ydn/hug-january-2011-kafka-presentation] > >> > * Kafka overview from Jakob Homan: [http://bit.ly/fLmoZz] > >> > * Kafka paper at NetDB 2011: [ > >> > > >> > > >> > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > >> > ] > >> > > >> > == Initial Source == > >> > Kafka has been under development at LinkedIn since November 2009. It > was > >> > open sourced by LinkedIn in January 2011. It is currently hosted on > >> github > >> > under the Apache license at [https://github.com/kafka-dev/kafka] > >> > > >> > Kafka is mainly written in Scala with some performance testing code in > >> > Java. > >> > Several clients have been contributed in other languages, including > >> Ruby, > >> > PHP, Clojure, .NET and Python. Its source tree is entirely self > >> contained > >> > and relies of simple build tool (sbt) as its build system and > dependency > >> > resolution mechanism. > >> > > >> > == External Dependencies == > >> > The dependencies all have Apache compatible licenses. > >> > > >> > == Cryptography == > >> > Not applicable. > >> > > >> > == Required Resources == > >> > === Mailing Lists === > >> > * kafka-private for private PMC discussions (with moderated > >> subscriptions) > >> > * kafka-dev * kafka-commits * kafka-user > >> > > >> > === Subversion Directory === > >> > [https://svn.apache.org/repos/asf/incubator/kafka] > >> > > >> > === Issue Tracking === > >> > JIRA Kafka (KAFKA) > >> > > >> > === Other Resources === > >> > The existing code already has unit tests, so we would like a Hudson > >> > instance > >> > to run them whenever a new patch is submitted. This can be added after > >> > project creation. > >> > > >> > == Initial Committers == > >> > * Jay Kreps > >> > * Jun Rao > >> > * Neha Narkhede > >> > * Jakob Homan > >> > > >> > == Affiliations == > >> > * Jay Kreps (LinkedIn) > >> > * Jun Rao (LinkedIn) > >> > * Neha Narkhede (LinkedIn) > >> > * Jakob Homan (LinkedIn) > >> > > >> > == Sponsors == > >> > === Champion === > >> > Chris Douglas (Apache Member) > >> > > >> > === Nominated Mentors === > >> > * Alan Cabrera (Apache Member) > >> > * Geir Magnusson, Jr. (Apache Member and Director) > >> > * Owen O'Malley (Apache Member) > >> > > >> > === Sponsoring Entity === > >> > We are requesting the Incubator to sponsor this project. > >> > > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org > For additional commands, e-mail: general-h...@incubator.apache.org > >
