Thanks for reply, David, your library is great and indeed the rebalancing is currently somewhat quirky and complicated. And I guess it doesn't make sense to implement it considering 0.9 is planned relatively soon.
On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <mum...@gmail.com> wrote: > Kane, > > I'm the creator of kafka-python, just thought I'd give some insight. > > Consumer rebalancing is actually pretty tricky to get right. It requires > interaction with ZooKeeper which (though possible via kazoo) is something > I've tried to avoid in kafka-python. It also seems a little strange to me > to mix your consumers between Java/Scala and Python. If you really need > rebalancing between Python consumers, you'd have to implement that on top > of kafka-python. > > Once the coordinator API is finalized for 0.9, I (or someone) will work on > implementing it in kafka-python > > Cheers > -David > > > On 10/1/13 11:56 AM, Kane Kane wrote: > >> The reason i was asking is that this library seems to have support only >> for >> SimpleConsumer >> https://github.com/mumrah/**kafka-python/<https://github.com/mumrah/kafka-python/>, >> i was curious if >> all should be implemented on client or kafka has some rebalancing logic >> and >> prevent consuming from the same queue on server side in case of >> SimpleConsumer api, but I see now that everything should implemented on >> the >> client side. >> >> Thanks. >> >> >> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <wangg...@gmail.com> wrote: >> >> I do not understand your question, what are you trying to implement? >>> >>> >>> On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <kane.ist...@gmail.com> wrote: >>> >>> So essentially you can't do "queue" pattern, unless you somehow >>>> implement >>>> locking on the client? >>>> >>>> >>>> On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <wangg...@gmail.com> >>>> >>> wrote: >>> >>>> SimpleConsumer do not have any concept of group management, only the >>>>> high-level consumers have. So multiple simple consumers can >>>>> >>>> independently >>> >>>> consume from the same partition(s). >>>>> >>>>> Guozhang >>>>> >>>>> >>>>> On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <kane.ist...@gmail.com> >>>>> >>>> wrote: >>> >>>> Yeah, I noticed that, i'm curious how balancing happens if >>>>>> >>>>> SimpleConsumer >>>> >>>>> is used. I.e. i can provide a partition to read from if i use >>>>>> SimpleConsumer, but what if someone else already attached to that >>>>>> partition, what would happen? Also what would happen if one >>>>>> >>>>> SimpleConsumer >>>>> >>>>>> attached to all partitions? Noone would be able to join? >>>>>> >>>>>> >>>>>> On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede < >>>>>> >>>>> neha.narkh...@gmail.com >>> >>>> wrote: >>>>>>> There are 2 types of consumer clients in Kafka - >>>>>>> >>>>>> ZookeeperConsumerConnector >>>>>> >>>>>>> and SimpleConsumer. Only the former has the re balancing logic. >>>>>>> >>>>>>> Thanks, >>>>>>> Neha >>>>>>> On Oct 1, 2013 6:30 AM, "Kane Kane" <kane.ist...@gmail.com> wrote: >>>>>>> >>>>>>> But it looks like some clients don't implement it? >>>>>>>> >>>>>>>> >>>>> >>>>> -- >>>>> -- Guozhang >>>>> >>>>> >>> >>> -- >>> -- Guozhang >>> >>> >
