I am ready to use low level.......... I do have a gevent worker server that can export that data I need to send as json. What about command line? I do over 10K qps can will be at 25K soon.
Thanks On Mon, Nov 11, 2013 at 4:25 AM, Marc Labbe <mrla...@gmail.com> wrote: > @Kane, Let's be clear, you could send multiple consumer requests in > parallel, but in terms of handling events, it would make things a little > harder to manage. E.g. the importance of reading messages in order, are you > ready to handle duplicates, ... For consumers, gevent may be useful when > handling events (store in db, forward to other processes, ...) but for > consumers purely reading from kafka, the model is pretty much to be blocked > until data becomes available, handle data, read the next batch, and so on. > > @David I think you are out of luck for out of the box options for gevent. > You may be able to do something if you are ready to use low level options. > > > On Sun, Nov 10, 2013 at 2:53 PM, David Montgomery < > davidmontgom...@gmail.com > > wrote: > > > I am using kafka0. 7.x I should mention and is nit by choice because > druid, > > the app I am using only supports 0.7.x. Brod is failing with gevent with > > 0.7.2. Brod totally does not work with gevent. > > > > Thanks > > > > > > > > > > > > On Mon, Nov 11, 2013 at 3:06 AM, Marc Labbe <mrla...@gmail.com> wrote: > > > > > Hi David, > > > > > > check for mahendra's fork of kafka-python, he has implemented gevent > > > support in a branch ( > > https://github.com/mahendra/kafka-python/tree/gevent) > > > but > > > it hasn't made it to the main repo (yet). I am not sure why and I > haven't > > > tested it myself. > > > > > > For brod, AFAIK, it doesn't support 0.8 so you have to use it with > 0.7.x. > > > On the other hand, kafka-python is the only python library I know of > that > > > supports 0.8. > > > > > > cheers, > > > marc > > > > > > > > > On Sat, Nov 9, 2013 at 6:34 PM, David Montgomery > > > <davidmontgom...@gmail.com>wrote: > > > > > > > Hi, > > > > > > > > Even even further...using/kafka-python ( > > > > https://github.com/mumrah/kafka-python) I get the below error. > There > > is > > > > nothing wrong with my kafka server. I can telnet to 9092 just fine. > > > > > > > > Again..hoping for a python client that works with gevent. > kafka-python > > > did > > > > nok event make it past the connection. > > > > > > > > kafka = KafkaClient(kafka_domain, 9092) > > > > No handlers could be found for logger "kafka" > > > > Traceback (most recent call last): > > > > File > "/home/ubuntu/workspace/rtbhui-devops/servers/worker_server.py", > > > > line 133, in <module> > > > > kafka = KafkaClient(kafka_domain, 9092) > > > > File > > > > > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py", > > > > line 32, in __init__ > > > > self._load_metadata_for_topics() > > > > File > > > > > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/kafka_python-0.8.1_1-py2.7.egg/kafka/client.py", > > > > line 69, in _load_metadata_for_topics > > > > raise Exception("All servers failed to process request") > > > > Exception: All servers failed to process request > > > > > > > > > > > > > > > > On Sun, Nov 10, 2013 at 5:47 AM, David Montgomery < > > > > davidmontgom...@gmail.com > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I asked this question on StackOverflow. No response. No solution > > > seems > > > > > tp work...even for a python lib that was designed for gevent and > > kafka, > > > > Is > > > > > there any otehr libary that works with gevent wit python that > works? > > > > > > > > > > > > > > > > > > > > > > > > > am trying to use gevent to write to kafka using brod on python 2.7. > > > > > > > > > > Here is the error message I get. Guess its to do blocking. brod > > > supports > > > > > tornado but I use gevent. > > > > > > > > > > No handlers could be found for logger "brod.socket"Traceback (most > > > > recent call last): > > > > > File "/var/chef/cache/src/gevent/gevent/greenlet.py", line 328, > in > > > run > > > > > result = self._run(*self.args, **self.kwargs) > > > > > File "worker_server.py", line 204, in execute_kafka_pipe > > > > > kafka.produce(topic,payload) > > > > > File > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/base.py", > > > > line 287, in produce > > > > > return self._write(request, callback) > > > > > File > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py", > > > > line 98, in _write > > > > > return self._write(data, callback, retries) > > > > > File > > > > > > > > > > "/usr/local/lib/python2.7/dist-packages/brod-0.3.2-py2.7.egg/brod/blocking.py", > > > > line 89, in _write > > > > > wrote_length += self._socket.send(data) > > > > > File "/var/chef/cache/src/gevent/gevent/socket.py", line 441, in > > send > > > > > self._wait(self._write_event) > > > > > File "/var/chef/cache/src/gevent/gevent/socket.py", line 292, in > > > _wait > > > > > assert watcher.callback is None, 'This socket is already used > by > > > > another greenlet: %r' % (watcher.callback, )AssertionError: This > socket > > > is > > > > already used by another greenlet: <bound method Waiter.switch of > > > > <gevent.hub.Waiter object at 0x1dece60>><Greenlet at 0x1d4b9b0: > > > > execute_kafka_pipe('topic-spend', '{"enode": 1, "city": "Cairns", > "dl": > > > > "en", "wnode)> failed with AssertionError > > > > > > > > > > I tried to use gevent-kakfa but depends on gevent-zookeeper. > > > > > > > > > > When trying to connect to zookeeper i get this message: > > > > > > > > > > Traceback (most recent call last): > > > > > File "/home/ubuntu/workspace/devops/servers/worker_server.py", > line > > > > 68, in <module> > > > > > framework = > gevent_zookeeper.ZookeeperFramework('localhost:2181', > > > 10) > > > > > File > > > > > "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/framework.py", > > > > line 241, in __init__ > > > > > self.client = ZookeeperClient(hosts, timeout) > > > > > File > > > > "/usr/local/lib/python2.7/dist-packages/gevent_zookeeper/client.py", > > line > > > > 211, in __init__ > > > > > self._event = gevent.core.event(AttributeError: 'module' object > > has > > > > no attribute 'core'Exception AttributeError: "'ZookeeperClient' > object > > > has > > > > no attribute '_event'" in <bound method ZookeeperClient.__del__ of > > > > <gevent_zookeeper.client.ZookeeperClient object at 0x274ded0>> > ignored > > > > > > > > > > Is there not a python lib that I can I write messages using gevent > > that > > > > > works? > > > > > > > > > > > > > > > > > > > >
