Which is the best ZK based implementation of kafka in node.js. Our use case is that a pool of node js http servers will be listening to clients which will send json over http. Using node js we'll do minimal decoration and compression (preferably snappy) and write to brokers. We might also need json to avro conversion but thats not a deal breaker. Consumers will be writing these events to S3 (to begin with we don't plan to maintain HDFS cluster). To begin with we'll have to support a peak load of 50K events / second, average being much less, around 2K events / second. Suggestions please. Is any one using franz-kafka in production. I'm only two days into kafka so don't know a lot, but franz-kafka looks clean and easy to work with.
If none of the existing node.js implementation is capable of this then we are willing to move to Scala or Java but node.js is the first choice. Thanks & Regards, Apoorva On Sat, Dec 22, 2012 at 2:25 AM, Radek Gruchalski < radek.gruchal...@portico.io> wrote: > We are using https://github.com/radekg/node-kafka, occasionally pushing > about 2500 messages, 3.5K each / second. No issues so far. Different story > with consumers. They are stable but under heavy load we experienced CPU > problems. I am the maintainer of that fork. The fork comes with ZK > integration. Another kafka module is this one: > https://github.com/dannycoates/franz-kafka. > > Kind regards, > Radek Gruchalski > radek.gruchal...@technicolor.com (mailto:radek.gruchal...@technicolor.com) > | radek.gruchal...@portico.io (mailto:radek.gruchal...@portico.io) | > ra...@gruchalski.com (mailto:ra...@gruchalski.com) > 00447889948663 > > Confidentiality: > This communication is intended for the above-named person and may be > confidential and/or legally privileged. > If it has come to you in error you must take no action based on it, nor > must you copy or show it to anyone; please delete/destroy and inform the > sender immediately. > > > On Thursday, 20 December 2012 at 18:31, Jun Rao wrote: > > > Chris, > > > > Not sure how stable those node.js clients are. In 0.8, we plan to > provide a > > native C version of the producer. A thin node.js layer can potentially be > > built on top of that. > > > > Thanks, > > > > Jun > > > > On Thu, Dec 20, 2012 at 8:46 AM, Christopher Alexander < > > calexan...@gravycard.com (mailto:calexan...@gravycard.com)> wrote: > > > > > During my due diligence to assess use of Kafka for both our activity > and > > > log message streams, I would like to ask the project committers and > > > community users about using Kafka with Node.js. Yes, I am aware that a > > > Kafka client exists for Node.js ( > > > https://github.com/marcuswestin/node-kafka), which has spurred further > > > interest by our front-end team. Here are my questions, excuse me if > they > > > seem "noobish". > > > > > > 1. How reliable is the Node.js client ( > > > https://github.com/marcuswestin/node-kafka) in production > applications? > > > If there are issues, what are they (the GitHub repo currently lists > none)? > > > 2. To support real-time activity streams within Node.js, what is the > > > recommended consumer polling interval? > > > 3. General advise observations on integrating a front-end based Node.js > > > application with Kafka mediated messaging. > > > > > > Thanks you! > > > > > > Chris > >
