I have a test data set of 1500 messages (~2.5 MB each) that I'm using to test Kafka throughput. I'm pushing this data using 40 Kafka producers, and I'm losing about 10% of the message on each trial.
I'm seeing errors of the following form: Failed to send producer request with correlation id 80 to broker 6 with data for partitions [kafka_end_test_3r,339] Produce request with correlation id 88 failed due to [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException Failed to send requests for topics kafka_end_test_3r with correlation ids in [15,77] My topic has 400 partitions and a replication factor of 3. I have a 9-node broker cluster. In the producer, I've set request.required.acks to -1, retry.backoff.ms to 500, message.send.max.retries to 20. What am I doing wrong? How can I make Kafka to accept 100% of messages? Any help would be greatly appreciated.
