Great idea Gwen! I think it would go a long way to making this work. Issue created :)
On Tue, Aug 19, 2014 at 4:33 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > Does it make sense to merge the Camus mailing list? (i.e. ask the > Camus community to merge?) Its a fairly large and popular client. > > On Tue, Aug 19, 2014 at 1:27 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > I also opened issues on 3 of the clients on github that I frequently > > use/involved in often enough.... would be great to get on the README as > > such. > > > > Thanks to the community for driving things along! > > > > /******************************************* > > Joe Stein > > Founder, Principal Consultant > > Big Data Open Source Security LLC > > http://www.stealth.ly > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > ********************************************/ > > > > > > On Tue, Aug 19, 2014 at 4:22 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > >> I just joined too, and tweeted. > >> > >> /******************************************* > >> Joe Stein > >> Founder, Principal Consultant > >> Big Data Open Source Security LLC > >> http://www.stealth.ly > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> ********************************************/ > >> > >> > >> On Tue, Aug 19, 2014 at 4:08 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> > >>> Cool. I just joined. I'll add it to the website so others can find it. > >>> If someone was willing to ping some of the other client developers and > >>> get them to join as well that would probably give us critical mass. > >>> > >>> -Jay > >>> > >>> On Tue, Aug 19, 2014 at 9:08 AM, Dana Powers <dana.pow...@rd.io> > wrote: > >>> > I created kafka-clie...@groups.google.com > >>> > > >>> > https://groups.google.com/forum/m/#!forum/kafka-clients > >>> > > >>> > No members and no guidelines yet, but it's a start. Would love to > get > >>> this > >>> > going. > >>> > > >>> > Dana > >>> > On Aug 19, 2014 9:03 AM, "Mark Roberts" <wiz...@gmail.com> wrote: > >>> > > >>> >> Did this mailing list ever get created? Was there consensus that it > >>> did or > >>> >> didn't need created? > >>> >> > >>> >> -Mark > >>> >> > >>> >> > On Jul 18, 2014, at 14:34, Jay Kreps <jay.kr...@gmail.com> wrote: > >>> >> > > >>> >> > A question was asked in another thread about what was an effective > >>> way > >>> >> > to contribute to the Kafka project for people who weren't very > >>> >> > enthusiastic about writing Java/Scala code. > >>> >> > > >>> >> > I wanted to kind of advocate for an area I think is really > important > >>> >> > and not as good as it could be--the client ecosystem. I think our > >>> goal > >>> >> > is to make Kafka effective as a general purpose, centralized, data > >>> >> > subscription system. This vision only really works if all your > >>> >> > applications, are able to integrate easily, whatever language they > >>> are > >>> >> > in. > >>> >> > > >>> >> > We have a number of pretty good non-java producers. We have been > >>> >> > lacking the features on the server-side to make writing non-java > >>> >> > consumers easy. We are fixing that right now as part of the > consumer > >>> >> > work going on right now (which moves a lot of the functionality in > >>> the > >>> >> > java consumer to the server side). > >>> >> > > >>> >> > But apart from this I think there may be a lot more we can do to > make > >>> >> > the client ecosystem better. > >>> >> > > >>> >> > Here are some concrete ideas. If anyone has additional ideas > please > >>> >> > reply to this thread and share them. If you are interested in > picking > >>> >> > any of these up, please do. > >>> >> > > >>> >> > 1. The most obvious way to improve the ecosystem is to help work > on > >>> >> > clients. This doesn't necessarily mean writing new clients, since > in > >>> >> > many cases we already have a client in a given language. I think > any > >>> >> > way we can incentivize fewer, better clients rather than many > >>> >> > half-working clients we should do. However we are working now on > the > >>> >> > server-side consumer co-ordination so it should now be possible to > >>> >> > write much simpler consumers. > >>> >> > > >>> >> > 2. It would be great if someone put together a mailing list just > for > >>> >> > client developers to share tips, tricks, problems, and so on. We > can > >>> >> > make sure all the main contributors on this too. I think this > could > >>> be > >>> >> > a forum for kind of directing improvements in this area. > >>> >> > > >>> >> > 3. Help improve the documentation on how to implement a client. We > >>> >> > have tried to make the protocol spec not just a dry document but > also > >>> >> > have it share best practices, rationale, and intentions. I think > this > >>> >> > could potentially be even better as there is really a range of > >>> options > >>> >> > from a very simple quick implementation to a more complex highly > >>> >> > optimized version. It would be good to really document some of the > >>> >> > options and tradeoffs. > >>> >> > > >>> >> > 4. Come up with a standard way of documenting the features of > >>> clients. > >>> >> > In an ideal world it would be possible to get the same information > >>> >> > (author, language, feature set, download link, source code, etc) > for > >>> >> > all clients. It would be great to standardize the documentation > for > >>> >> > the client as well. For example having one or two basic examples > that > >>> >> > are repeated for every client in a standardized way. This would > let > >>> >> > someone come to the Kafka site who is not a java developer, and > click > >>> >> > on the link for their language and view examples of interacting > with > >>> >> > Kafka in the language they know using the client they would > >>> eventually > >>> >> > use. > >>> >> > > >>> >> > 5. Build a Kafka Client Compatibility Kit (KCCK) :-) The idea is > >>> this: > >>> >> > anyone who wants to implement a client would implement a simple > >>> >> > command line program with a set of standardized options. The > >>> >> > compatibility kit would be a standard set of scripts that ran > their > >>> >> > client using this command line driver and validate its behavior. > E.g. > >>> >> > for a producer it would test that it correctly can send messages, > >>> that > >>> >> > the ordering is retained, that the client correctly handles > >>> >> > reconnection and metadata refresh, and compression. The output > would > >>> >> > be a list of features that passed are certified, and perhaps basic > >>> >> > performance information. This would be an easy way to help client > >>> >> > developers write correct clients, as well as having a standardized > >>> >> > comparison for the clients that says that they work correctly. > >>> >> > > >>> >> > -Jay > >>> >> > >>> > >> > >> >
