You need to first decide the conditions that need to be met for you to scale to 50 consumers. These can be as simple as the consumer lag. Look at the console offset checker tool and see if any of those numbers make sense. Your existing consumers could also produce some metrics based on which another process will decide when to spawn new customers.
-- Sharninder On Wed, Apr 29, 2015 at 11:58 PM, Nimi Wariboko Jr <n...@channelmeter.com> wrote: > Hi, > > I was wondering what options there are/what other people are doing for > horizontally scaling kafka consumers? Basically if I have 100 partitions > and 10 consumers, and want to temporarily scale up to 50 consumers, what > can I do? > > So far I've thought of just simply tracking consumer membership somehow > (either through zookeeper's ephemeral nodes or maybe using gossip) on the > consumers to achieve consensus on who consumes what. Another option would > be having a router, possibly using something like nsq (I understand that > they are similar pieces of software, but what we are going for is a > persistent distributed queue (sharding) which is why I'm looking into > Kafka)? > -- -- Sharninder
