In the old producer yes, in the new producer (available in 0.8.1.1) the batch size is by bytes instead of #. messages, which gives you a better control.
Guozhang On Sat, Jun 7, 2014 at 2:48 PM, Kane Kane <kane.ist...@gmail.com> wrote: > Ah, that makes sense then. As I understand right now there is no > something like queue.buffering.max.size to control batch size? > > On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > With compression, the batch of messages are compressed in to a "single" > > wrapper message and sent to the broker, and the broker will reject the > > request if this single message's size is larger than 1MB. So you need to > > either change your max request size on broker or reduce your producer > batch > > size. > > > > Guozhang > > > > > > On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <kane.ist...@gmail.com> wrote: > > > >> Yes, messages were compressed with gzip and I've enabled the same > >> compression in mirrormaker producer. > >> > >> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > >> > Kane, > >> > > >> > Did you use any compression method? > >> > > >> > Guozhang > >> > > >> > > >> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <kane.ist...@gmail.com> > wrote: > >> > > >> >> I've tried to run mirrormaker tools in async mode and I get > >> >> WARN Produce request with correlation id 263 failed due to > >> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException > >> >> (kafka.producer.async.DefaultEventHandler) > >> >> > >> >> I don't get error in sync mode. My message.max.bytes is default > >> >> (1000000). As I understand the difference between sync and async mode > >> >> is that async collects batches before sending. Does it mean that > batch > >> >> should be less than message.max.bytes, or any single message within > >> >> batch should be less than that. > >> >> > >> >> Thanks. > >> >> > >> > > >> > > >> > > >> > -- > >> > -- Guozhang > >> > > > > > > > > -- > > -- Guozhang > -- -- Guozhang
