What version of Kafka are you using? Have you increased the max message size on the broker (default to 1MB)?
Thanks, Jun On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <kyleban...@gmail.com> wrote: > I have a test data set of 1500 messages (~2.5 MB each) that I'm using to > test Kafka throughput. I'm pushing this data using 40 Kafka producers, and > I'm losing about 10% of the message on each trial. > > I'm seeing errors of the following form: > > Failed to send producer request with correlation id 80 to broker 6 with > data for partitions [kafka_end_test_3r,339] > Produce request with correlation id 88 failed due to > [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException > Failed to send requests for topics kafka_end_test_3r with correlation ids > in [15,77] > > My topic has 400 partitions and a replication factor of 3. I have a 9-node > broker cluster. > > In the producer, I've set request.required.acks to -1, retry.backoff.ms to > 500, message.send.max.retries to 20. > > What am I doing wrong? How can I make Kafka to accept 100% of messages? Any > help would be greatly appreciated. >
