I am unable to reproduce it.
I did note also that all the consumer offsets reset in this application,
not just the streams, so it appears that whatever happened is not
streams-specific. The only reason I can think of for all the consumers to
do this, is that the committed offsets information was "lost" somehow, and
so when the service started back up, it reverted to "earliest" as per the
configuration of "auto.offset.reset". In a test I ran, the logging output
and behavior of the consumers matches exactly this scenario.
Now what I'm trying to understand is under what conditions the committed
offsets would be "lost"? The only ones I can think of are:
a) The consumers were idle for longer than "offsets.retention.minutes"
(default of 7 days in our env, and no this was not the case for us)
b) Somebody mistakenly blew away the data in the topic where Kafka stores
the consumer offsets (as far as I know, this didn't happen, but we don't
have ACLs implemented -- what logs can I check for?)
What other possibilities are there?
Also, are there any other situations other than the committed offsets not
being present, in which the Java consumer Fetcher may print the log message
"Resetting offset for partition {} to offset {}."?
Regards,
Raman
On Thu, Feb 21, 2019 at 2:00 AM Matthias J. Sax <matth...@confluent.io>
wrote:
> Thanks for reporting the issue!
>
> Are you able to reproduce it? If yes, can you maybe provide broker and
> client logs in DEBUG level?
>
> -Matthias
>
> On 2/20/19 7:07 PM, Raman Gupta wrote:
> > I have an exactly-once stream that reads a topic, transforms it, and
> writes
> > new messages into the same topic as well as other topics. I am using
> Kafka
> > 2.1.0. The stream applications run in Kubernetes.
> >
> > I did a k8s deployment of the application with minor changes to the code
> --
> > absolutely no changes to anything related to business logic or streams,
> and
> > no changes to the brokers at all. For some reason when the streams
> > application restarted, it reset a bunch of offsets to very old values,
> and
> > started re-processing old messages again (definitely violating the
> exactly
> > once principle!).
> >
> > There is no explanation in the logs at all as to why this offset reset
> > happened, including in the broker logs, and I am at a loss to understand
> > what is going on.
> >
> > Some example logs:
> >
> > February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,630 INFO --- [b0f59-StreamThread-1]
> > org.apa.kaf.cli.con.int.Fetcher : [Consumer
> >
> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer,
> > groupId=prod-cisSegmenter] Resetting offset for partition
> > prod-file-events-5 to offset 224.
> > February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,630 INFO --- [b0f59-StreamThread-1]
> > org.apa.kaf.cli.con.int.Fetcher : [Consumer
> >
> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer,
> > groupId=prod-cisSegmenter] Resetting offset for partition
> > prod-file-events-2 to offset 146.
> > February 20th 2019, 19:53:23.623 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,623 INFO --- [b0f59-StreamThread-1]
> org.apa.kaf.str.KafkaStreams
> > : stream-client
> > [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59] State transition
> > from REBALANCING to RUNNING
> > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,622 INFO --- [b0f59-StreamThread-1]
> > org.apa.kaf.cli.con.KafkaConsumer : [Consumer
> >
> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer,
> > groupId=] Unsubscribed all topics or patterns and assigned partitions
> > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,622 INFO --- [b0f59-StreamThread-1]
> > org.apa.kaf.cli.con.KafkaConsumer : [Consumer
> >
> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer,
> > groupId=] Unsubscribed all topics or patterns and assigned partitions
> > February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21
> > 00:53:23,622 INFO --- [b0f59-StreamThread-1]
> > org.apa.kaf.str.pro.int.StreamThread : stream-thread
> > [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1]
> > State transition from PARTITIONS_ASSIGNED to RUNNING
> >
> > Unless I'm totally misunderstanding something about how consumer groups
> > offsets are supposed to work, this behaviour is very very wrong.
> >
> > Regards,
> > Raman
> >
>
>
