You're right Stevo, I should re-phrase to say that there can be no more _active_ consumers than there are partitions (within a single consumer group). I'm guessing that's what Nimi is alluding to asking, but perhaps he can elaborate on whether he's using consumer groups and/or whether the 100 partitions are all for a single topic, or multiple topics.
On 29 April 2015 at 13:38, Stevo Slavić <ssla...@gmail.com> wrote: > Please correct me if wrong, but I think it is really not hard constraint > that one cannot have more consumers (from same group) than partitions on > single topic - all the surplus consumers will not be assigned to consume > any partition, but they can be there and as soon as one active consumer > from same group goes offline (its connection to ZK is dropped), consumers > from the group will be rebalanced so one passively waiting consumer will > become active. > > Kind regards, > Stevo Slavic. > > On Wed, Apr 29, 2015 at 2:25 PM, David Corley <davidcor...@gmail.com> > wrote: > > > If the 100 partitions are all for the same topic, you can have up to 100 > > consumers working as part of a single consumer group for that topic. > > You cannot have more consumers than there are partitions within a given > > consumer group. > > > > On 29 April 2015 at 08:41, Nimi Wariboko Jr <n...@channelmeter.com> > wrote: > > > > > Hi, > > > > > > I was wondering what options there are for horizontally scaling kafka > > > consumers? Basically if I have 100 partitions and 10 consumers, and > want > > to > > > temporarily scale up to 50 consumers, what options do I have? > > > > > > So far I've thought of just simply tracking consumer membership somehow > > > (either through Raft or zookeeper's znodes) on the consumers. > > > > > >
