Hi Joel, Any updates on the c++ producer?
Thanks, Anand On 3 April 2013 05:59, Joel Koshy <jjkosh...@gmail.com> wrote: > Yes - we would be interested in doing that. I have been spending most of my > time over the past couple weeks on the C++ library (currently, only for the > producer). It is reasonably stable, although it has not been tried and > tested in production. > > I can start with publishing a wiki describing the design/implementation and > list out future work that others can contribute to if there is interest, > but I'll quickly summarize some of its goals here: > - Use non-blocking I/O for sending requests/receiving responses. This > mitigates the request-response RTT latency that the 0.8 protocol introduces > (if acks != 0). > - C-compliant API, so that other languages e.g., Ruby/Python can wrap the > library. I understand there are people working on 0.8 clients in those > languages so this is just an alternative approach. > - Non-blocking variants for all operations in the API: there are some > use-cases where blocking in the user-thread is unacceptable. > - Keep third-party libraries to a minimum to make porting to other > platforms easier. > > There are two options in pursuing this: > 1 - Contribute it as part of the Kafka project. i.e., re-introduce the > clients directory. We removed it because none of the committers were > original authors of any of the non-Java clients and did not have the > bandwidth/background to review patches for those clients. > 2 - Maintain it externally. > I was going to also suggest "sub-project" as a third option, but I heard > from Jakob that it is no longer encouraged by the board and a client by > itself would not be a substantial sub-project. > > If we want to do (1), we would obviously go through the standard > contribution process: i.e., create a jira for it and provide a patch that > people can review; and other implementations will also be considered if > contributors provide patches. > > (2) is also fine, although one benefit with (1) is that it would be > reasonable to use the existing infrastructure (mailing lists, issue > tracking) for the Kafka project. > > Any preferences/thoughts? > > Thanks, > > Joel > > On Mon, Apr 1, 2013 at 8:22 AM, mrevilgnome <mrevilgn...@gmail.com> wrote: > > > Any interest in open sourcing it now and picking up contributors? > > > > > > On Mon, Apr 1, 2013 at 7:54 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > At LinkedIn, we are also building a native C producer client for 0.8. > It > > > uses non-blocking socket I/O to improve the producer throughput. We > plan > > to > > > open source it when it's fully tested, hopefully in a couple of months. > > > > > > Thanks, > > > > > > Jun > > > > > > On Wed, Mar 27, 2013 at 8:48 PM, Matthew Stump < > mst...@matthewstump.com > > > >wrote: > > > > > > > Howdy, > > > > > > > > I'm considering the use of Kafka in the rewrite of a big legacy > > product. > > > A > > > > good chunk of the back end code is going to be written in C++ (large > in > > > > memory data-structures). The two possible options available to me for > > > > clients appear to be: > > > > > > > > https://github.com/edenhill/librdkafka > > > > > > > > and > > > > > > > > https://github.com/quipo/kafka-cpp > > > > > > > > The problem is that librdkafka currently only works on Linux due to > the > > > use > > > > of the AF_NETLINK API, and thread local storage. There may be other > > > issues, > > > > but I just started playing with it today and that's what I've > > discovered > > > > thus far. > > > > > > > > kafka-cpp is incomplete (no consumer) and it looks unused. > > > > > > > > For either I would need to hop in and do some significant work. Is > > there > > > > any client I'm missing that can shorten my path? > > > > > > > > If I adopt one of these projects (lets say kafka-cpp) am I better off > > > > implementing the 0.8 protocol? I'de like to have something running in > > > > staging a couple months from now. How far out is 0.8? > > > > > > > > Thanks, > > > > Matt Stump > > > > > > > > > >
