All right, looks like I was able to reproduce this after all. I believe I
had to restart the producer after changing the offset retention setting in
the broker. Its quite surprising to me that this was the default behavior
before Kafka 2.1.0 -- I imagine quite a few people have been bitten hard by
Hmm, it turns out I was mistaken -- we are running Kafka 2.0.0, not 2.1.0.
We are on 2.1.0 on the client side, not the server.
Reading through KAFKA-4682, am I right in understanding that prior to
2.1.0, the consumer group offsets could be deleted if the consumers are
running, but idle for longer
You can read the `__consumer_offsets` topics directly to see if the
offset are there or not:
https://stackoverflow.com/questions/33925866/kafka-how-to-read-from-consumer-offsets-topic
Also, if your brokers are on version 2.1, offsets should not be deleted
as long as the consumer group is online. T
I am unable to reproduce it.
I did note also that all the consumer offsets reset in this application,
not just the streams, so it appears that whatever happened is not
streams-specific. The only reason I can think of for all the consumers to
do this, is that the committed offsets information was "
Thanks for reporting the issue!
Are you able to reproduce it? If yes, can you maybe provide broker and
client logs in DEBUG level?
-Matthias
On 2/20/19 7:07 PM, Raman Gupta wrote:
> I have an exactly-once stream that reads a topic, transforms it, and writes
> new messages into the same topic as
I have an exactly-once stream that reads a topic, transforms it, and writes
new messages into the same topic as well as other topics. I am using Kafka
2.1.0. The stream applications run in Kubernetes.
I did a k8s deployment of the application with minor changes to the code --
absolutely no changes
