To be honest, 0.8 it's a pretty big change from how things are currently managed in samsa. I don't have much of an ETA at the moment, but I'm hoping to find the time in the next month or two.
Keith. On Thu, Oct 3, 2013 at 11:59 AM, Kane Kane <kane.ist...@gmail.com> wrote: > Hi Keith, thanks for update! Interestingly i've found your library > yesterday and was going to borrow your zookeeper code for partition > management. Do you have any ETA for the 0.8 support? > > Thanks! > > > On Thu, Oct 3, 2013 at 8:09 AM, Keith Bourgoin <keith.bourg...@gmail.com > >wrote: > > > Hi Kane, > > > > I just wanted to chime in as well. I currently maintain samsa ( > > https://github.com/getsamsa/samsa or PyPI), which does implement the > > rebalancing logic, but currently doesn't support 0.8. If you're on 0.7.x > > still, it might be helpful. If not, we're working on 0.8 support, but it > > probably won't be ready for a bit. > > > > Cheers, > > > > Keith. > > > > > > On Tue, Oct 1, 2013 at 1:39 PM, Kane Kane <kane.ist...@gmail.com> wrote: > > > > > Thanks for reply, David, your library is great and indeed the > rebalancing > > > is currently somewhat quirky and complicated. And I guess it doesn't > make > > > sense to implement it considering 0.9 is planned relatively soon. > > > > > > > > > On Tue, Oct 1, 2013 at 10:09 AM, David Arthur <mum...@gmail.com> > wrote: > > > > > > > Kane, > > > > > > > > I'm the creator of kafka-python, just thought I'd give some insight. > > > > > > > > Consumer rebalancing is actually pretty tricky to get right. It > > requires > > > > interaction with ZooKeeper which (though possible via kazoo) is > > something > > > > I've tried to avoid in kafka-python. It also seems a little strange > to > > me > > > > to mix your consumers between Java/Scala and Python. If you really > need > > > > rebalancing between Python consumers, you'd have to implement that on > > top > > > > of kafka-python. > > > > > > > > Once the coordinator API is finalized for 0.9, I (or someone) will > work > > > on > > > > implementing it in kafka-python > > > > > > > > Cheers > > > > -David > > > > > > > > > > > > On 10/1/13 11:56 AM, Kane Kane wrote: > > > > > > > >> The reason i was asking is that this library seems to have support > > only > > > >> for > > > >> SimpleConsumer https://github.com/mumrah/**kafka-python/< > > > https://github.com/mumrah/kafka-python/>, > > > >> i was curious if > > > >> all should be implemented on client or kafka has some rebalancing > > logic > > > >> and > > > >> prevent consuming from the same queue on server side in case of > > > >> SimpleConsumer api, but I see now that everything should implemented > > on > > > >> the > > > >> client side. > > > >> > > > >> Thanks. > > > >> > > > >> > > > >> On Tue, Oct 1, 2013 at 8:52 AM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > >> > > > >> I do not understand your question, what are you trying to > implement? > > > >>> > > > >>> > > > >>> On Tue, Oct 1, 2013 at 8:42 AM, Kane Kane <kane.ist...@gmail.com> > > > wrote: > > > >>> > > > >>> So essentially you can't do "queue" pattern, unless you somehow > > > >>>> implement > > > >>>> locking on the client? > > > >>>> > > > >>>> > > > >>>> On Tue, Oct 1, 2013 at 8:35 AM, Guozhang Wang <wangg...@gmail.com > > > > > >>>> > > > >>> wrote: > > > >>> > > > >>>> SimpleConsumer do not have any concept of group management, only > the > > > >>>>> high-level consumers have. So multiple simple consumers can > > > >>>>> > > > >>>> independently > > > >>> > > > >>>> consume from the same partition(s). > > > >>>>> > > > >>>>> Guozhang > > > >>>>> > > > >>>>> > > > >>>>> On Tue, Oct 1, 2013 at 8:11 AM, Kane Kane <kane.ist...@gmail.com > > > > > >>>>> > > > >>>> wrote: > > > >>> > > > >>>> Yeah, I noticed that, i'm curious how balancing happens if > > > >>>>>> > > > >>>>> SimpleConsumer > > > >>>> > > > >>>>> is used. I.e. i can provide a partition to read from if i use > > > >>>>>> SimpleConsumer, but what if someone else already attached to > that > > > >>>>>> partition, what would happen? Also what would happen if one > > > >>>>>> > > > >>>>> SimpleConsumer > > > >>>>> > > > >>>>>> attached to all partitions? Noone would be able to join? > > > >>>>>> > > > >>>>>> > > > >>>>>> On Tue, Oct 1, 2013 at 6:33 AM, Neha Narkhede < > > > >>>>>> > > > >>>>> neha.narkh...@gmail.com > > > >>> > > > >>>> wrote: > > > >>>>>>> There are 2 types of consumer clients in Kafka - > > > >>>>>>> > > > >>>>>> ZookeeperConsumerConnector > > > >>>>>> > > > >>>>>>> and SimpleConsumer. Only the former has the re balancing logic. > > > >>>>>>> > > > >>>>>>> Thanks, > > > >>>>>>> Neha > > > >>>>>>> On Oct 1, 2013 6:30 AM, "Kane Kane" <kane.ist...@gmail.com> > > wrote: > > > >>>>>>> > > > >>>>>>> But it looks like some clients don't implement it? > > > >>>>>>>> > > > >>>>>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> -- Guozhang > > > >>>>> > > > >>>>> > > > >>> > > > >>> -- > > > >>> -- Guozhang > > > >>> > > > >>> > > > > > > > > > >
