Hi Abhimanyu, What is your offsets.retention.minutes on brokers set to? It could have happened that the offsets in __consumer_offsets topic got wiped out after the configured time and so when the consumer restarted, it did not find where to start with. If auto.offset.reset is set to earliest on consumers, you will end up re-consuming your messages.
This is something on the same lines: https://issues.apache.org/jira/browse/KAFKA-3806 HTH. On Fri, May 19, 2017 at 12:14 AM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM, > 1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition > in which messages contains the total of 3 Million messages. It took 5 days > to completely process all the messages and as soon as message got processed > i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the > topic I stopped the consumer .but after 6 hrs again it was showing the lag > of 2 million message which I found that were duplicate messages. This thing > is happening very frequently. My offsets are stored on Kafka broker itself. > My server configuration is: > > broker.id=1 > delete.topic.enable=true > #listeners=PLAINTEXT://:9092 > #advertised.listeners=PLAINTEXT://your.host.name:9092 > num.network.threads=3 > num.io.threads=8 > socket.send.buffer.bytes=102400 > socket.receive.buffer.bytes=102400 > socket.request.max.bytes=104857600 > log.dirs=/kafka/data/logs > num.partitions=1 > num.recovery.threads.per.data.dir=5 > log.flush.interval.messages=10000 > #log.flush.interval.ms=1000 > log.retention.hours=480 > log.retention.bytes=1073741824 > log.segment.bytes=1073741824 > log.retention.check.interval.ms=300000 > zookeeper.connect=<zkIp>:2181 > zookeeper.connection.timeout.ms=6000 > > Is there in the configuration that I am missing? > Any help is appreciated >
