you can configure max.request.size
<https://kafka.apache.org/documentation/#configuration> when you create
Kafka producer, about the second questions, Where is the key size + value
size + header size calculated? you can check `
org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound()
{...}
`
Ty
On Fri, Jul 6, 2018 at 12:42 AM, Jerry Richardson <
jerryrichard...@tutanota.com> wrote:
> My administrator will not allow messages larger than 1MB to be stored in
> Kafka.
> How can I limit the size of my messages to 1MB? If I have a message larger
> than 1MB, I want to truncate or throw away the message to avoid the
> RecordTooLargeException. What is the max size of the headers? Where is
> the key size + value size + header size calculated?
>
> 5. Jul 2018 14:15 by jiangtao....@zuora.com <mailto:jiangtao....@zuora.com
> >.INVALID:
>
>
> > sorry, just saw your email a little late.
> >
> > I am confused with why you said `I cannot increase the max size of
> messages
> > stored in Kafka` ? can we add some explanation ?
> >
> > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson <
> > jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>> >
> wrote:
> >
> >>
> >> What class is this in? What's the maximum header size? Is there
> >> documentation on this?
> >>
> >> 2. Jul 2018 18:29 by >> jiangtao....@zuora.com <mailto:
> jiangtao....@zuora.com>>> <>> mailto:jiangtao....@zuora.com <mailto:
> mailto:jiangtao....@zuora.com>
> >> >.INVALID:
> >>
> >>
> >> > You can consider increase `max.request.size` a little big (the
> default
> >> > value is `1048576`.), after checking Kafka client source code, they
> count
> >> > [`key size` + `value size` + `header size` + others] together, so it's
> >> > possible the calculated size is a little bigger than the default
> value.
> >> >
> >> > please check with > >> https://kafka.apache.org <
> https://kafka.apache.org/>
> >> documentation/#configuration <>> https://kafka.apache.org <
> https://kafka.apache.org/>
> >> documentation/#configuration>> .
> >> >
> >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> jerryrichard...@tutanota.com
> <mailto:jerryrichard...@tutanota.com>>> <mailto:
> >> jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>>>
> >> > wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> I get this error even when my records are smaller than the 1000012
> byte
> >> >> limit:
> >> >>
> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request
> >> >> included a message larger than the max message size the server will
> >> accept.
> >> >>
> >> >> How do I ensure that my producer doesn't send records that are too
> >> large?
> >> >>
> >> >> Thanks in advance for any suggestions and help.
> >> >>
> >>
>
