Hi Nitay, What version were you running before the upgrade? And was this a broker upgrade? I would suggest filing a JIRA ticket, including as much information as possible and ideally consumer and broker logs from the period where the issue happened.
Ismael On Sun, Apr 5, 2020, 10:13 AM Nitay Kufert <nita...@ironsrc.com> wrote: > Hey, > > We are using kafka streams across our tech stack for the last 2-3 years. > > Lately, we moved to kafka 2.4.1 and we started noticing a troubling > behaviour that we didn't see before: > > Without apparent reason, a specific partition on a specific consumer loses > its offset, with the following messages appearing on the consumer: > >> >> [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4] >> Fetch offset 1222791071 is out of range for partition SomeTopic-48, >> resetting offset: >> > [Consumer clientId=consumer-XConsumerGroup4-2, groupId=XConsumerGroup4] >> Resetting offset for partition SomeTopic-48 to offset 1216430527.: > > > from the broker, the only thing i could find is: > >> BrokerId 1033 is no longer a coordinator for the group XConsumerGroup4. >> Proceeding cleanup for other alive groups >> >> > Another way to see the same thing, from our monitoring (DD): > [image: Screen Shot 2020-04-05 at 16.32.19.png] > > The recovery you are seeing is after I run partition offset reset manually > (using kafka-consumer-groups.sh --bootstrap-server localhost:9092 --topic > SomeTopic:57 --group XConsumerGroup4 --reset-offsets --to-datetime > '2020-04-02T07:00:00.000') > > Any idea what can be causing this? we have it happen to us at least 5 > times since the upgrade, and before that i don't remember it even happening > to us. > > Let me know if you need more data from me > Thanks > -- > > Nitay Kufert > Backend Team Leader > [image: ironSource] <http://www.ironsrc.com> > > email nita...@ironsrc.com > mobile +972-54-5480021 > fax +972-77-5448273 > skype nitay.kufert.ssa > 121 Menachem Begin St., Tel Aviv, Israel > ironsrc.com <http://www.ironsrc.com> > [image: linkedin] <https://www.linkedin.com/company/ironsource> [image: > twitter] <https://twitter.com/ironsource> [image: facebook] > <https://www.facebook.com/ironSource> [image: googleplus] > <https://plus.google.com/+ironsrc> > This email (including any attachments) is for the sole use of the intended > recipient and may contain confidential information which may be protected > by legal privilege. If you are not the intended recipient, or the employee > or agent responsible for delivering it to the intended recipient, you are > hereby notified that any use, dissemination, distribution or copying of > this communication and/or its content is strictly prohibited. If you are > not the intended recipient, please immediately notify us by reply email or > by telephone, delete this email and destroy any copies. Thank you. >
