Ah, that makes sense then. As I understand right now there is no something like queue.buffering.max.size to control batch size?
On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wangg...@gmail.com> wrote: > With compression, the batch of messages are compressed in to a "single" > wrapper message and sent to the broker, and the broker will reject the > request if this single message's size is larger than 1MB. So you need to > either change your max request size on broker or reduce your producer batch > size. > > Guozhang > > > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <kane.ist...@gmail.com> wrote: > >> Yes, messages were compressed with gzip and I've enabled the same >> compression in mirrormaker producer. >> >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> > Kane, >> > >> > Did you use any compression method? >> > >> > Guozhang >> > >> > >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <kane.ist...@gmail.com> wrote: >> > >> >> I've tried to run mirrormaker tools in async mode and I get >> >> WARN Produce request with correlation id 263 failed due to >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException >> >> (kafka.producer.async.DefaultEventHandler) >> >> >> >> I don't get error in sync mode. My message.max.bytes is default >> >> (1000000). As I understand the difference between sync and async mode >> >> is that async collects batches before sending. Does it mean that batch >> >> should be less than message.max.bytes, or any single message within >> >> batch should be less than that. >> >> >> >> Thanks. >> >> >> > >> > >> > >> > -- >> > -- Guozhang >> > > > > -- > -- Guozhang
