Re: Balancing based on lag

2016-11-26 Thread Ewen Cheslack-Postava
Jens, Sorry, I'm very late to this thread but figure it might be worth following up since I think this is a cool feature of the new consumer but isn't well known. You actually have *quite* a bit of flexibility in controlling how partition assignment happens. The key hook is the partition.assignmen

Re: Balancing based on lag

2016-09-12 Thread Jens Rantil
Hi Stevo, Thank you for your response. Yes, I understand there can only be one active consumer per partition and I understand that partitions should be spread evenly. However, there will almost always be cases when they are somewhat unbalanced. That said, how consumers are distributed among parti

Re: Balancing based on lag

2016-09-11 Thread Stevo Slavić
Hello Jens, By design there can be only one active consumer per consumer group per partion at a time, only one thread after being assigned (acquiring a lock) moves the offset for consumer group for partition, so no concurrency problems. Typically that assignment lasts long until rebalancing gets t