Key issue with gevent is there can be no C bindings. If pure python then the sockets can be monkey patched as long as pure python code. I use gevent to run redis-py to make async calls to redis even though the client in nature is blocking. I do believe your client is pure python?
Thanks On Sun, Feb 17, 2013 at 1:15 PM, David Arthur <mum...@gmail.com> wrote: > Replying to both messages inline: > > > On 2/16/13 9:07 PM, David Montgomery wrote: > >> By the way..I assume that python-kafka is gevent safe? >> > No idea, I've never used greenlet/gevent before. If the question is "are > you doing anything unusual in kafka-python", then the answer is no. Just > basic method calls and some socket stuff > > >> Thanks >> >> >> On Sun, Feb 17, 2013 at 10:04 AM, David Montgomery < >> davidmontgom...@gmail.com> wrote: >> >> Ok...great...I see now about offsets. I see how I can manage on >>> restarts. Thanks! >>> >>> So...considering I am in a disposable machine world then I will consider >>> redis as a centralized store. Makes sense? >>> >> You can certainly used Redis as a fast, in-memory queue. It is, of > course, an entirely different system with different design goals and > features. The applicability of Redis or Kafka depends on your use case. > > >>> What is the time frame for v8 release? >>> >> I believe it is farily imminent, maybe sometime in March? > >> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Sun, Feb 17, 2013 at 3:27 AM, David Arthur <mum...@gmail.com> wrote: >>> >>> Greetings! >>>> >>>> I am the maintainer of kafka-python. Very cool to see it used in the >>>> wild. >>>> >>>> The kafka-python library supports the low-level protocols of Kafka 0.7 >>>> (Produce/Fetch/MultiProduce/****MultiFetch). When you ask Kafka for >>>> >>>> messages via a Fetch request, you specify an offset + range (much like >>>> reading a file). The `iter_messages` helper returns an iterator that >>>> automatically handles paging offsets through successive Fetch requests. >>>> However, it does not support _saving_ your offsets. One of the >>>> parameters >>>> to iter_messages is the offset to start at, so when you re-run your >>>> script >>>> it will start at that point again. >>>> >>>> In 0.7, clients must talk to ZooKeeper in order to persist offsets in a >>>> Kafka-compatible way (or they could just save them locally depending on >>>> the >>>> use case). Talking to ZooKeeper from Python is somewhat troublesome, and >>>> implementing the Kafka "consumer group rebalancing" is even more >>>> troublesome - so I chose to omit it. >>>> >>>> In 0.8 (not yet released), consumer offsets are managed centrally by the >>>> Kafka brokers and have APIs for clients to commit and fetch offsets. I >>>> am >>>> in the process of implementing a 0.8 compatible version of kafka-python. >>>> >>>> So for the time being, you are on your own with regards to offset >>>> management :-/ >>>> >>>> Cheers! >>>> >>>> -David >>>> >>>> >>>> On 2/16/13 1:35 PM, Philip O'Toole wrote: >>>> >>>> You need to read the Kafka design docs. Kafka does not delete messages >>>>> just because a Consumer reads it. It does not track what messages have >>>>> been >>>>> consumed by any Consumer. >>>>> >>>>> It is up to Consumers to start off where they left off, by always >>>>> asking >>>>> for the right message (via offsets). >>>>> >>>>> Philip >>>>> >>>>> On Feb 16, 2013, at 4:48 AM, David Montgomery < >>>>> davidmontgom...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>>> I have a zookeer and kafka set up. >>>>>> >>>>>> I am using this python client: https://github.com/mumrah/** >>>>>> kafka-python >>>>>> <https://github.com/mumrah/**kafka-python<https://github.com/mumrah/kafka-python> >>>>>> > >>>>>> >>>>>> >>>>>> I can send and receive messages but they are not deleted. >>>>>> >>>>>> How can I send a message to kafka and no other consumer can use it? >>>>>> >>>>>> >>>>>> I feel I am missing something on how kafka works >>>>>> >>>>>> def produce(): >>>>>> kafka = KafkaClient("xxx.xxx", 9092) >>>>>> kafka.send_messages_simple("****my-topic", "some message") >>>>>> >>>>>> kafka.close() >>>>>> print 'done' >>>>>> >>>>>> def consume(): >>>>>> kafka = KafkaClient("xxx.xxx", 9092) >>>>>> for msg in kafka.iter_messages("my-topic"****, 0, 0, >>>>>> >>>>>> 1024*1024,False): >>>>>> print(msg.payload) >>>>>> kafka.close() >>>>>> print 'done' >>>>>> >>>>>> Every time I ran the above...everytime I ran consume the messages just >>>>>> grew >>>>>> from previous messages. >>>>>> >>>>>> Am I missing something on the server.properties file? >>>>>> >>>>>> Thanks >>>>>> >>>>>> >
