I'm using a (java) consumer with default configuration, so auto-commit is enabled. The consumer is reading from 5 partitions of a single topic. The consumer processes one message at a time (synchronously). Sometimes, large numbers of messages are posted to the topic, and the consumer will have to work for one or more hours to process all messages. This is not a problem.
Sometimes, if there is a significant lag, I notice that the committed offsets that Kafka reports for the consumer, are very strange, or even impossible. It goes up and down. I've created a graph for the current-offset and log-end-offset for one partition: https://ibb.co/frJ3eb. The dotted line is the log-end-offset (seems fine), and the solid line is the current-offset, as reported by the kafka-consumer-groups.sh tool. As you can clearly see, the current offset suddenly drops by a few thousand, then gradually increases, but shows a number of spikes, until it reaches the original value again, then suddenly increases twice by a few thousand. Obviously the current offset should never decrease, but also the sudden increases are also impossible; the consumer is technically not able to process the messages this fast. The same behaviour is shown for the other partitions of this consumer, and also for other consumers of the same topic. I'm using Kafka 0.10.0.1 and kafka-client 0.10.0.1. Does anyone have an idea on what is happening here? Regards, Tom
