Hi all, I am using Kafka 2.11-0.10.0.1 and Zookeeper 3.4.8. I have a cluster of 4 servers(A,B,C,D) running one kafka broker on each of them and, one zookeeper server on server A. Data is initially produced from server A using a Kafka Producer and it goes through servers B,C,D being subjected to processing and finally reaches server A again(gets consumed using a Kafka Consumer).
Topics created on the end of each process has 2 partitions with a replication-factor of 3. Other configurations include, unclean.leader.election.enable=false acks=all retries=0 I let the producer run for a while in server A, then kill one of the Kafka brokers on the cluster(B,C,D) while data processing takes place and restart it. When consuming from the end of server A, I notice a considerable amount of data lost which varies on each run! ex:- on an input of 1 million events 5930 events are lost. Is the reason for this the Kafka Producer not guaranteeing Exactly-once processing or is this due to some other reason? What other reasons cause data loss?
