Thanks Gwen. When you're saying that I can add consumers to the same group, does that also hold true if those consumers are running on different machines? Or in different JVMs?
-- Sharninder On Wed, Oct 8, 2014 at 11:35 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > If you use the high level consumer implementation, and register all > consumers as part of the same group - they will load-balance > automatically. > > When you add a consumer to the group, if there are enough partitions > in the topic, some of the partitions will be assigned to the new > consumer. > When a consumer crashes, once its node in ZK times out, other > consumers will get its partitions. > > Gwen > > On Wed, Oct 8, 2014 at 10:39 AM, Sharninder <sharnin...@gmail.com> wrote: > > Hi, > > > > I'm not even sure if this is a valid use-case, but I really wanted to run > > it by you guys. How do I load balance my consumers? For example, if my > > consumer machine is under load, I'd like to spin up another VM with > another > > consumer process to keep reading messages off any topic. On similar > lines, > > how do you guys handle consumer failures? Suppose one consumer process > gets > > an exception and crashes, is it possible for me to somehow make sure that > > there is another process that is still reading the queue for me? > > > > -- > > Sharninder >
