Hi, While testing message delivery using kafka, I realized that few duplicate messages got delivered by the consumers in the same consumer group (two consumers got the same message with few milli-seconds difference). However, I do not see any redundancy at the producer or broker. One more observation is that - this is not happening when I use only one consumer thread.
I am running 3 brokers (0.8.2.1) with 3 Zookeeper nodes. There are 3 partitions in the topic and replication-factor is 3. For producing, am using New Producer with compression.type=none. On the consumer end, I have 3 High level consumers in the same consumer group running with one consumer thread each, on three different hosts. Auto commit is set to true for consumer. Size of each message would range anywhere between 0.7 KB and 2 MB. The max volume for this test is 100 messages/hr. I looked at controller log for any possibility of consumer rebalance during this time, but did not find any. In the server log of all the brokers the error - java.io.IOException: Connection reset by peer is almost being written continuously. So, is it possible to achieve exactly-once delivery with the current high level consumer without needing an extra layer to remove redundancy? Could you please point me to any settings or logs that would help me tune the configuration ? *PS: I tried searching for similar discussions, but could not find any. If its already been answered, please provide the link. Thanks, Kris
