Re: Getting my Kafka code example working out of the box

Hi, I was trying the single producer/consumer model via http://kafka.apache.org/documentation.html#quick-start. The code was from https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor the producer. I'll try to repeat all the steps again in case I missed something but this wa

Re: Getting my Kafka code example working out of the box

Yes, I have 1 consumer running with 1 partition; or at least i think i have 1 partition. I ran the cmd from the documentation to create the topic *bin/kafka-create-topic.sh --zookeeper localhost:2181 --replica 1 --partition 1 --topic test* and I ran the console consumer command later, as specifi

CSV Metrics reporter possible duplicate metrics filenames

HI, I sometimes use the CSV Metrics reporter to get some metrics from Kafka with more accuracy and w/o the hassle of having to configure our central metrics system to fetch the metrics from JMX, especially for one-off tests. However, every time I start a broker, I get errors from kafka like this

Re: Purgatory

I've just updated the wiki with most of your comments hoping to make it more clear. I stole much of Joel's comment so all the credit goes to him if it works! The line between documenting the code and describing it is always thin. My intention was not to make anything official at first either, hence

Re: Getting my Kafka code example working out of the box

If you have 5 partitions and 3 consumers in one consumer group, the consumers will balance all 5 partitions such that 2 consumers will get data from 2 partitions each and 1 consumer will get data from remaining 5th partition. If you have 3 partitions and 5 consumers in one consumer group, 3 consume

Re: Getting my Kafka code example working out of the box

Do you have enough consumers representing the number of partitions you have? You should have one consumer within a consumer group for each partition... Think of it as threads, if you have 5 partitions you need 5 threads... if you only have 3 consumers(threads) then it is possible the data is on on

Re: Getting my Kafka code example working out of the box

Thanks for everyone's reply and trying to help me. I think I got the test producer program to work but I don't know why. Again, using the test code found at https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Examplefor the producer straight from the website didn't work. However, wh

Re: How to use gevent with python

I am ready to use low level.. I do have a gevent worker server that can export that data I need to send as json. What about command line? I do over 10K qps can will be at 25K soon. Thanks On Mon, Nov 11, 2013 at 4:25 AM, Marc Labbe wrote: > @Kane, Let's be clear, you could send mult

Re: How to use gevent with python

@Kane, Let's be clear, you could send multiple consumer requests in parallel, but in terms of handling events, it would make things a little harder to manage. E.g. the importance of reading messages in order, are you ready to handle duplicates, ... For consumers, gevent may be useful when handling

Re: How to use gevent with python

I am using kafka0. 7.x I should mention and is nit by choice because druid, the app I am using only supports 0.7.x. Brod is failing with gevent with 0.7.2. Brod totally does not work with gevent. Thanks On Mon, Nov 11, 2013 at 3:06 AM, Marc Labbe wrote: > Hi David, > > check for mahendra's

Re: How to use gevent with python

New client rewrite proposal includes async Producer, but not async Consumer, i think there is a reason. You can not send new consume request before previous one is finished? On Sun, Nov 10, 2013 at 11:42 AM, Marc Labbe wrote: > Kane, you can probably achieve async consumer using the client direct

Re: How to use gevent with python

Kane, you can probably achieve async consumer using the client directly (kafka.client.send_fetch_request) which a parameter to pass in a callback. I haven't used it myself. On Sun, Nov 10, 2013 at 2:33 PM, Marc Labbe wrote: > I was just browsing in the issues list and found the original issue f

Re: How to use gevent with python

I was just browsing in the issues list and found the original issue from mahendra for gevent support, https://github.com/mumrah/kafka-python/issues/37. I suggest moving the discussion there. On Sun, Nov 10, 2013 at 2:06 PM, Marc Labbe wrote: > Hi David, > > check for mahendra's fork of kafka-py

Re: How to use gevent with python

How it's possible to have async consumer? On Sun, Nov 10, 2013 at 11:06 AM, Marc Labbe wrote: > Hi David, > > check for mahendra's fork of kafka-python, he has implemented gevent > support in a branch (https://github.com/mahendra/kafka-python/tree/gevent) but > it hasn't made it to the main repo

Re: How to use gevent with python

Hi David, check for mahendra's fork of kafka-python, he has implemented gevent support in a branch (https://github.com/mahendra/kafka-python/tree/gevent) but it hasn't made it to the main repo (yet). I am not sure why and I haven't tested it myself. For brod, AFAIK, it doesn't support 0.8 so you

Re: Purgatory

Marc, thanks much for documenting the guts! There is one correction for Fetch Request handling: When is it satisfied? The fetch size requested is reached - ie. the amount of data the consumer wishes to receive in one response Consumer configuration: *fetch.message.max.bytes* As per the code: