You can read the `__consumer_offsets` topics directly to see if the offset are there or not: https://stackoverflow.com/questions/33925866/kafka-how-to-read-from-consumer-offsets-topic
Also, if your brokers are on version 2.1, offsets should not be deleted as long as the consumer group is online. The `offset.retention.minutes` config starts to tick when the consumer groups goes offline (cf https://issues.apache.org/jira/browse/KAFKA-4682) -Matthias On 2/21/19 3:08 PM, Raman Gupta wrote: > I am unable to reproduce it. > > I did note also that all the consumer offsets reset in this application, > not just the streams, so it appears that whatever happened is not > streams-specific. The only reason I can think of for all the consumers to > do this, is that the committed offsets information was "lost" somehow, and > so when the service started back up, it reverted to "earliest" as per the > configuration of "auto.offset.reset". In a test I ran, the logging output > and behavior of the consumers matches exactly this scenario. > > Now what I'm trying to understand is under what conditions the committed > offsets would be "lost"? The only ones I can think of are: > > a) The consumers were idle for longer than "offsets.retention.minutes" > (default of 7 days in our env, and no this was not the case for us) > b) Somebody mistakenly blew away the data in the topic where Kafka stores > the consumer offsets (as far as I know, this didn't happen, but we don't > have ACLs implemented -- what logs can I check for?) > > What other possibilities are there? > > Also, are there any other situations other than the committed offsets not > being present, in which the Java consumer Fetcher may print the log message > "Resetting offset for partition {} to offset {}."? > > Regards, > Raman > > > On Thu, Feb 21, 2019 at 2:00 AM Matthias J. Sax <matth...@confluent.io> > wrote: > >> Thanks for reporting the issue! >> >> Are you able to reproduce it? If yes, can you maybe provide broker and >> client logs in DEBUG level? >> >> -Matthias >> >> On 2/20/19 7:07 PM, Raman Gupta wrote: >>> I have an exactly-once stream that reads a topic, transforms it, and >> writes >>> new messages into the same topic as well as other topics. I am using >> Kafka >>> 2.1.0. The stream applications run in Kubernetes. >>> >>> I did a k8s deployment of the application with minor changes to the code >> -- >>> absolutely no changes to anything related to business logic or streams, >> and >>> no changes to the brokers at all. For some reason when the streams >>> application restarted, it reset a bunch of offsets to very old values, >> and >>> started re-processing old messages again (definitely violating the >> exactly >>> once principle!). >>> >>> There is no explanation in the logs at all as to why this offset reset >>> happened, including in the broker logs, and I am at a loss to understand >>> what is going on. >>> >>> Some example logs: >>> >>> February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,630 INFO --- [b0f59-StreamThread-1] >>> org.apa.kaf.cli.con.int.Fetcher : [Consumer >>> >> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer, >>> groupId=prod-cisSegmenter] Resetting offset for partition >>> prod-file-events-5 to offset 224. >>> February 20th 2019, 19:53:23.630 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,630 INFO --- [b0f59-StreamThread-1] >>> org.apa.kaf.cli.con.int.Fetcher : [Consumer >>> >> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-consumer, >>> groupId=prod-cisSegmenter] Resetting offset for partition >>> prod-file-events-2 to offset 146. >>> February 20th 2019, 19:53:23.623 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,623 INFO --- [b0f59-StreamThread-1] >> org.apa.kaf.str.KafkaStreams >>> : stream-client >>> [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59] State transition >>> from REBALANCING to RUNNING >>> February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,622 INFO --- [b0f59-StreamThread-1] >>> org.apa.kaf.cli.con.KafkaConsumer : [Consumer >>> >> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer, >>> groupId=] Unsubscribed all topics or patterns and assigned partitions >>> February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,622 INFO --- [b0f59-StreamThread-1] >>> org.apa.kaf.cli.con.KafkaConsumer : [Consumer >>> >> clientId=prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1-restore-consumer, >>> groupId=] Unsubscribed all topics or patterns and assigned partitions >>> February 20th 2019, 19:53:23.622 cis-69b4bc6fb7-8xnhc 2019-02-21 >>> 00:53:23,622 INFO --- [b0f59-StreamThread-1] >>> org.apa.kaf.str.pro.int.StreamThread : stream-thread >>> [prod-cisSegmenter-abeba614-3bda-44bc-bc48-a278de9b0f59-StreamThread-1] >>> State transition from PARTITIONS_ASSIGNED to RUNNING >>> >>> Unless I'm totally misunderstanding something about how consumer groups >>> offsets are supposed to work, this behaviour is very very wrong. >>> >>> Regards, >>> Raman >>> >> >> >
signature.asc
Description: OpenPGP digital signature
