Thanks so much, Jun. That seems to have fixed the problem. I increased both message.max.bytes and replica.fetch.max.bytes on the broker.
For the benefit of future Kafka users, how hard would it be to build out some clearer error messaging for this case? On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <jun...@gmail.com> wrote: > Also, don't forget to increase replica.fetch.max.bytes to be larger than > the max message size. > > Thanks, > > Jun > > On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <jun...@gmail.com> wrote: > > > What version of Kafka are you using? Have you increased the max message > > size on the broker (default to 1MB)? > > > > Thanks, > > > > Jun > > > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <kyleban...@gmail.com> > wrote: > > > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using to > >> test Kafka throughput. I'm pushing this data using 40 Kafka producers, > and > >> I'm losing about 10% of the message on each trial. > >> > >> I'm seeing errors of the following form: > >> > >> Failed to send producer request with correlation id 80 to broker 6 with > >> data for partitions [kafka_end_test_3r,339] > >> Produce request with correlation id 88 failed due to > >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException > >> Failed to send requests for topics kafka_end_test_3r with correlation > ids > >> in [15,77] > >> > >> My topic has 400 partitions and a replication factor of 3. I have a > 9-node > >> broker cluster. > >> > >> In the producer, I've set request.required.acks to -1, retry.backoff.ms > >> to > >> 500, message.send.max.retries to 20. > >> > >> What am I doing wrong? How can I make Kafka to accept 100% of messages? > >> Any > >> help would be greatly appreciated. > >> > > > > >
