Re: [EXTERNAL] - Lost messages and messed up offsets

2017-11-30 Thread Tom van den Berge
If I understand correctly, the "auto.offset.reset" setting is only used if there is no offset available in Kafka (i.e. no offset has ever be committed?), or if the offset does not exist anymore. In my situation, I don't understand how either situation would be possible. The consumers continuously c

RE: [EXTERNAL] - Lost messages and messed up offsets

2017-11-29 Thread Isabelle Giguère
Hi; With default configuration, your consumers are set with auto.offset.reset=latest So on restart, the consumers start to read the offset of 0 minutes ago, not the offset of 30 minutes ago (or whatever the lag was). https://kafka.apache.org/documentation/#configuration auto.offset.reset What to