I am using single node Kafka broker (v 0.10.2) Hardware (8 core, 16 GB RAM,
1 TD HardDisk) and zookeeper (v 3.4.8). I have a topic with 200 partition
in which messages contains the total of 3 Million messages. It took 5 days
to completely process all the messages and as soon as message got processed
i.e. Kafka-consumer-groups.sh showed 0 lag in all the partition of the
topic I stopped the consumer .but after 6 hrs again it was showing the lag
of 2 million message which I found that were duplicate messages. This thing
is happening very frequently. My offsets are stored on Kafka broker itself.
My server configuration is:
broker.id=1
delete.topic.enable=true
#listeners=PLAINTEXT://:9092
#advertised.listeners=PLAINTEXT://your.host.name:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/kafka/data/logs
num.partitions=1
num.recovery.threads.per.data.dir=5
log.flush.interval.messages=10000
#log.flush.interval.ms=1000
log.retention.hours=480
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=<zkIp>:2181
zookeeper.connection.timeout.ms=6000
Is there in the configuration that I am missing?
Any help is appreciated
