Kyle, We have developed a new (pure java) producer in trunk. It should have better error messaging. Could you give it a try and see if it points out the problem clearer to you?
Thanks, Jun On Tue, Sep 23, 2014 at 8:30 PM, Kyle Banker <kyleban...@gmail.com> wrote: > Thanks so much, Jun. That seems to have fixed the problem. I increased both > message.max.bytes and replica.fetch.max.bytes on the broker. > > For the benefit of future Kafka users, how hard would it be to build out > some clearer error messaging for this case? > > On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <jun...@gmail.com> wrote: > > > Also, don't forget to increase replica.fetch.max.bytes to be larger than > > the max message size. > > > > Thanks, > > > > Jun > > > > On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > What version of Kafka are you using? Have you increased the max message > > > size on the broker (default to 1MB)? > > > > > > Thanks, > > > > > > Jun > > > > > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <kyleban...@gmail.com> > > wrote: > > > > > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using > to > > >> test Kafka throughput. I'm pushing this data using 40 Kafka producers, > > and > > >> I'm losing about 10% of the message on each trial. > > >> > > >> I'm seeing errors of the following form: > > >> > > >> Failed to send producer request with correlation id 80 to broker 6 > with > > >> data for partitions [kafka_end_test_3r,339] > > >> Produce request with correlation id 88 failed due to > > >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException > > >> Failed to send requests for topics kafka_end_test_3r with correlation > > ids > > >> in [15,77] > > >> > > >> My topic has 400 partitions and a replication factor of 3. I have a > > 9-node > > >> broker cluster. > > >> > > >> In the producer, I've set request.required.acks to -1, > retry.backoff.ms > > >> to > > >> 500, message.send.max.retries to 20. > > >> > > >> What am I doing wrong? How can I make Kafka to accept 100% of > messages? > > >> Any > > >> help would be greatly appreciated. > > >> > > > > > > > > >
