Jason, Glad that you asked. This is only an issue if compression is turned on for a batch of messages. The problem is that the compressed batch is treated as a single message, whose size has to be smaller than max.message.bytes. If you are sending data uncompressed, only each uncompressed message needs to be less than max.message.bytes. The batch size doesn't really matter in this case.
Thanks, Jun On Thu, Sep 26, 2013 at 8:11 PM, Jason Rosenberg <j...@squareup.com> wrote: > I've been doing some testing, trying to understand how the > max.message.bytes works, with respect to sending batches of messages. In a > previous discussion, there appeared to be a suggestion that one work around > when triggering a MessageSizeTooLargeException is to reduce the batch size > and resubmit the batch in smaller sub-batches. > > However, so far in my testing, I'm not seeing a way to trigger a > MessageSizeTooLargeException by having a large batch of messages, whose > cumulative size is greater than max.message.bytes, but without any > individual message exceeding the max. And in fact, I'm able to send > through some very large batches. E.g. 200 messages of 500000 bytes each in > a single batch (where the max.message.bytes is 1000000. However, if > anyone of the messages in the batch exceeds that limit, then it will reject > the whole batch. > > Is this the expected behavior? What was being referred to in the previous > discussions around retrying smaller batch sizes as a work-around for > MessageSizeTooLargeException. > > I haven't been using compression with any of these tests, does that make a > difference? > > Thanks, > > Jason >
