I have an exactly-once stream that reads a topic, transforms it, and writes
new messages into the same topic as well as other topics. I am using Kafka
2.1.0. The stream applications run in Kubernetes.
I did a k8s deployment of the application with minor changes to the code --
absolutely no changes to anything related to business logic or streams, and
no changes to the brokers at all. For some reason when the streams
application restarted, it reset a bunch of offsets to very old values, and
started re-processing old messages again (definitely violating the exactly
once principle!).
There is no explanation in the logs at all as to why this offset reset
happened, including in the broker logs, and I am at a loss to understand
what is going on.
Some example logs:
February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,630 INFO --- [b0f59-StreamThread-1]
org.apa.kaf.cli.con.int.Fetcher : [Consumer
clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer,
groupId=prod-cisSegmenter] Resetting offset for partition
prod-file-events-5 to offset 224.
February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,630 INFO --- [b0f59-StreamThread-1]
org.apa.kaf.cli.con.int.Fetcher : [Consumer
clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer,
groupId=prod-cisSegmenter] Resetting offset for partition
prod-file-events-2 to offset 146.
February 20th 2019, 19:53:23.623 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,623 INFO --- [b0f59-StreamThread-1] org.apa.kaf.str.KafkaStreams
: stream-client
[prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59] State transition
from REBALANCING to RUNNING
February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,622 INFO --- [b0f59-StreamThread-1]
org.apa.kaf.cli.con.KafkaConsumer : [Consumer
clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer,
groupId=] Unsubscribed all topics or patterns and assigned partitions
February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,622 INFO --- [b0f59-StreamThread-1]
org.apa.kaf.cli.con.KafkaConsumer : [Consumer
clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer,
groupId=] Unsubscribed all topics or patterns and assigned partitions
February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
00:53:23,622 INFO --- [b0f59-StreamThread-1]
org.apa.kaf.str.pro.int.StreamThread : stream-thread
[prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1]
State transition from PARTITIONS_ASSIGNED to RUNNING
Unless I'm totally misunderstanding something about how consumer groups
offsets are supposed to work, this behaviour is very very wrong.
Regards,
Raman
