Also, don't forget to increase replica.fetch.max.bytes to be larger than the max message size.
Thanks, Jun On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <jun...@gmail.com> wrote: > What version of Kafka are you using? Have you increased the max message > size on the broker (default to 1MB)? > > Thanks, > > Jun > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <kyleban...@gmail.com> wrote: > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using to >> test Kafka throughput. I'm pushing this data using 40 Kafka producers, and >> I'm losing about 10% of the message on each trial. >> >> I'm seeing errors of the following form: >> >> Failed to send producer request with correlation id 80 to broker 6 with >> data for partitions [kafka_end_test_3r,339] >> Produce request with correlation id 88 failed due to >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException >> Failed to send requests for topics kafka_end_test_3r with correlation ids >> in [15,77] >> >> My topic has 400 partitions and a replication factor of 3. I have a 9-node >> broker cluster. >> >> In the producer, I've set request.required.acks to -1, retry.backoff.ms >> to >> 500, message.send.max.retries to 20. >> >> What am I doing wrong? How can I make Kafka to accept 100% of messages? >> Any >> help would be greatly appreciated. >> > >
