So...if you have a consumer running on A, and then you start another one on B, I would think it would trigger a rebalance. I am not 100% sure whether A would just keep being the active consumer or if it would switch to B. That being said, yes, I think the general idea is correct.
In terms of offsets, yes, there is a chance you could get duplicate messages when you switch between consumers. We manually commit offsets after each message to help mitigate the problem, but it is still possible to receive duplicates. Your best bet is to make your consumer idempotent so that it can handle those duplicates. Hope that helps. Thanks, Subhash On Tue, Feb 13, 2018 at 10:44 AM, Xavier Noria <f...@hashref.com> wrote: > On Tue, Feb 13, 2018 at 2:59 PM, Subhash Sriram <subhash.sri...@gmail.com> > wrote: > > Hey Xavier, > > > > Within a consumer group, you can only have as many consumers as you have > > partitions in a topic. If you start more consumers than partitions within > > the same group, they will just be idle. > > > > Excellent, I am reading the docs but cannot anticipate some scenarios with > certainty. > > I am thinking about situations involving two machines, for fault-tolerance, > deploy procedures without downtime, etc. > > Let's say we have machines A and B, running consumers that belong to the > same consumer group. > > I guess the broker is then going to send messages only to the one that > opened the TCP connection first, right? > > I correct, let's assume it is A. The broker is going to accept a connection > from B, but there are no more partitions to balance, so B is idle and > sending heartbeats just fine. If there is a network issue between the > broker and A, and the socket of B is up, Kafka is then going to assign the > unique partition to B automatically right? > > In that case, if the hiccup is solved one minute later and the connection > in A is restored, then A is going to become the idle instance? > > If that is all correct, the only issue I foresee is committing offsets. If > the client commits offsets every 10 seconds and the network cut happens, I > believe the instance taking over is going to receive duplicated messages in > the general case, right? >
