Thanks for reporting the issue! Are you able to reproduce it? If yes, can you maybe provide broker and client logs in DEBUG level?
-Matthias On 2/20/19 7:07 PM, Raman Gupta wrote: > I have an exactly-once stream that reads a topic, transforms it, and writes > new messages into the same topic as well as other topics. I am using Kafka > 2.1.0. The stream applications run in Kubernetes. > > I did a k8s deployment of the application with minor changes to the code -- > absolutely no changes to anything related to business logic or streams, and > no changes to the brokers at all. For some reason when the streams > application restarted, it reset a bunch of offsets to very old values, and > started re-processing old messages again (definitely violating the exactly > once principle!). > > There is no explanation in the logs at all as to why this offset reset > happened, including in the broker logs, and I am at a loss to understand > what is going on. > > Some example logs: > > February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,630 INFO --- [b0f59-StreamThread-1] > org.apa.kaf.cli.con.int.Fetcher : [Consumer > clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer, > groupId=prod-cisSegmenter] Resetting offset for partition > prod-file-events-5 to offset 224. > February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,630 INFO --- [b0f59-StreamThread-1] > org.apa.kaf.cli.con.int.Fetcher : [Consumer > clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer, > groupId=prod-cisSegmenter] Resetting offset for partition > prod-file-events-2 to offset 146. > February 20th 2019, 19:53:23.623 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,623 INFO --- [b0f59-StreamThread-1] org.apa.kaf.str.KafkaStreams > : stream-client > [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59] State transition > from REBALANCING to RUNNING > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,622 INFO --- [b0f59-StreamThread-1] > org.apa.kaf.cli.con.KafkaConsumer : [Consumer > clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer, > groupId=] Unsubscribed all topics or patterns and assigned partitions > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,622 INFO --- [b0f59-StreamThread-1] > org.apa.kaf.cli.con.KafkaConsumer : [Consumer > clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer, > groupId=] Unsubscribed all topics or patterns and assigned partitions > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 > 00:53:23,622 INFO --- [b0f59-StreamThread-1] > org.apa.kaf.str.pro.int.StreamThread : stream-thread > [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1] > State transition from PARTITIONS_ASSIGNED to RUNNING > > Unless I'm totally misunderstanding something about how consumer groups > offsets are supposed to work, this behaviour is very very wrong. > > Regards, > Raman >
signature.asc
Description: OpenPGP digital signature
