Ah, that makes sense then. As I understand right now there is no
something like queue.buffering.max.size to control batch size?

On Sat, Jun 7, 2014 at 2:32 PM, Guozhang Wang <wangg...@gmail.com> wrote:
> With compression, the batch of messages are compressed in to a "single"
> wrapper message and sent to the broker, and the broker will reject the
> request if this single message's size is larger than 1MB. So you need to
> either change your max request size on broker or reduce your producer batch
> size.
>
> Guozhang
>
>
> On Sat, Jun 7, 2014 at 2:09 PM, Kane Kane <kane.ist...@gmail.com> wrote:
>
>> Yes, messages were compressed with gzip and I've enabled the same
>> compression in mirrormaker producer.
>>
>> On Sat, Jun 7, 2014 at 12:56 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>> > Kane,
>> >
>> > Did you use any compression method?
>> >
>> > Guozhang
>> >
>> >
>> > On Fri, Jun 6, 2014 at 2:15 PM, Kane Kane <kane.ist...@gmail.com> wrote:
>> >
>> >> I've tried to run mirrormaker tools in async mode and I get
>> >> WARN Produce request with correlation id 263 failed due to
>> >> [benchmark1,30]: kafka.common.MessageSizeTooLargeException
>> >> (kafka.producer.async.DefaultEventHandler)
>> >>
>> >> I don't get error in sync mode. My message.max.bytes is default
>> >> (1000000). As I understand the difference between sync and async mode
>> >> is that async collects batches before sending. Does it mean that batch
>> >> should be less than message.max.bytes, or any single message within
>> >> batch should be less than that.
>> >>
>> >> Thanks.
>> >>
>> >
>> >
>> >
>> > --
>> > -- Guozhang
>>
>
>
>
> --
> -- Guozhang

Reply via email to