Consumers will rebalance if you add partitions, add consumers to the group or if a consumer leaves the group.
Consumers will leave the group after not communicating with the server for a period set by session.timeout.ms. This is usually due to an exception in the code polling with the consumer, or message processing code taking too long. If your consumers are reprocessing messages thus causing emails to send, it implies that they weren't able to commit their offsets before failing/timing out. We had a similar issue in a database sink that consumed from Kafka and duplicated data because it took too long, and hit the session timeout, and then wasn't able to commits its offsets. So I'd look closely at your consuming code and log every possible source of exceptions. Kind regards, Liam Clarke On Fri, 24 May 2019, 7:37 pm ASHOK MACHERLA, <iash...@outlook.com> wrote: > Dear Team Member > > Currently we are using Kafka 0.10.1, zookeeper 3.4.6 versions. In our > project we have to send bulk emails to customers for this purpose we are > using Kafka cluster setup. > > But customers are getting same emails for roughly 30-40 times. This is > very worst thing. In this situation our consumer group is showing > rebalancing. Might be its could be reason ????? > Currently one topic we are using for this. We have 10 partitions and 10 > consumers. > I hope we have enough partitions and consumer as well. > But I don’t know exactly number of partitions & consumer are required to > overcome this issue. > > Can you please suggest us to fix this issue. > > If anything changes required in Kafka side as well as consumer side?? > How to stop rebalancing issue?????? > Please suggest us, Thanks > > > > Sent from Outlook. > >
