@Kane, Let's be clear, you could send multiple consumer requests in parallel, but in terms of handling events, it would make things a little harder to manage. E.g. the importance of reading messages in order, are you ready to handle duplicates, ... For consumers, gevent may be useful when handling events (store in db, forward to other processes, ...) but for consumers purely reading from kafka, the model is pretty much to be blocked until data becomes available, handle data, read the next batch, and so on.
@David I think you are out of luck for out of the box options for gevent. You may be able to do something if you are ready to use low level options. On Sun, Nov 10, 2013 at 2:53 PM, David Montgomery <davidmontgom...@gmail.com > wrote: > I am using kafka0. 7.x I should mention and is nit by choice because druid, > the app I am using only supports 0.7.x. Brod is failing with gevent with > 0.7.2. Brod totally does not work with gevent. > > Thanks > > > > > > On Mon, Nov 11, 2013 at 3:06 AM, Marc Labbe <mrla...@gmail.com> wrote: > > > Hi David, > > > > check for mahendra's fork of kafka-python, he has implemented gevent > > support in a branch ( > https://github.com/mahendra/kafka-python/tree/gevent) > > but > > it hasn't made it to the main repo (yet). I am not sure why and I haven't > > tested it myself. > > > > For brod, AFAIK, it doesn't support 0.8 so you have to use it with 0.7.x. > > On the other hand, kafka-python is the only python library I know of that > > supports 0.8. > > > > cheers, > > marc > > > > > > On Sat, Nov 9, 2013 at 6:34 PM, David Montgomery > > <davidmontgom...@gmail.com>wrote: > > > > > Hi, > > > > > > Even even further...using/kafka-python ( > > > https://github.com/mumrah/kafka-python) I get the below error. There > is > > > nothing wrong with my kafka server. I can telnet to 9092 just fine. > > > > > > Again..hoping for a python client that works with gevent. kafka-python > > did > > > nok event make it past the connection. > > > > > > kafka = KafkaClient(kafka_domain, 9092) > > > No handlers could be found for logger "kafka" > > > Traceback (most recent call last): > > > File "/home/ubuntu/workspace/rtbhui-devops/servers/worker_server.py", > > > line 133, in <module> > > > kafka = KafkaClient(kafka_domain, 9092) > > > File > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py", > > > line 32, in __init__ > > > self._load_metadata_for_topics() > > > File > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py", > > > line 69, in _load_metadata_for_topics > > > raise Exception("All servers failed to process request") > > > Exception: All servers failed to process request > > > > > > > > > > > > On Sun, Nov 10, 2013 at 5:47 AM, David Montgomery < > > > davidmontgom...@gmail.com > > > > wrote: > > > > > > > Hi, > > > > > > > > I asked this question on StackOverflow. No response. No solution > > seems > > > > tp work...even for a python lib that was designed for gevent and > kafka, > > > Is > > > > there any otehr libary that works with gevent wit python that works? > > > > > > > > > > > > > > > > > > > > am trying to use gevent to write to kafka using brod on python 2.7. > > > > > > > > Here is the error message I get. Guess its to do blocking. brod > > supports > > > > tornado but I use gevent. > > > > > > > > No handlers could be found for logger "brod.socket"Traceback (most > > > recent call last): > > > > File "/var/chef/cache/src/gevent/gevent/greenlet.py", line 328, in > > run > > > > result = self._run(*self.args, **self.kwargs) > > > > File "worker_server.py", line 204, in execute_kafka_pipe > > > > kafka.produce(topic,payload) > > > > File > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/base.py", > > > line 287, in produce > > > > return self._write(request, callback) > > > > File > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py", > > > line 98, in _write > > > > return self._write(data, callback, retries) > > > > File > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py", > > > line 89, in _write > > > > wrote_length += self._socket.send(data) > > > > File "/var/chef/cache/src/gevent/gevent/socket.py", line 441, in > send > > > > self._wait(self._write_event) > > > > File "/var/chef/cache/src/gevent/gevent/socket.py", line 292, in > > _wait > > > > assert watcher.callback is None, 'This socket is already used by > > > another greenlet: %r' % (watcher.callback, )AssertionError: This socket > > is > > > already used by another greenlet: <bound method Waiter.switch of > > > <gevent.hub.Waiter object at 0x1dece60>><Greenlet at 0x1d4b9b0: > > > execute_kafka_pipe('topic-spend', '{"enode": 1, "city": "Cairns", "dl": > > > "en", "wnode)> failed with AssertionError > > > > > > > > I tried to use gevent-kakfa but depends on gevent-zookeeper. > > > > > > > > When trying to connect to zookeeper i get this message: > > > > > > > > Traceback (most recent call last): > > > > File "/home/ubuntu/workspace/devops/servers/worker_server.py", line > > > 68, in <module> > > > > framework = gevent_zookeeper.ZookeeperFramework('localhost:2181', > > 10) > > > > File > > > "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/framework.py", > > > line 241, in __init__ > > > > self.client = ZookeeperClient(hosts, timeout) > > > > File > > > "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/client.py", > line > > > 211, in __init__ > > > > self._event = gevent.core.event(AttributeError: 'module' object > has > > > no attribute 'core'Exception AttributeError: "'ZookeeperClient' object > > has > > > no attribute '_event'" in <bound method ZookeeperClient.__del__ of > > > <gevent_zookeeper.client.ZookeeperClient object at 0x274ded0>> ignored > > > > > > > > Is there not a python lib that I can I write messages using gevent > that > > > > works? > > > > > > > > > > > > > >
