My administrator will not allow messages larger than 1MB to be stored in Kafka. How can I limit the size of my messages to 1MB? If I have a message larger than 1MB, I want to truncate or throw away the message to avoid the RecordTooLargeException. What is the max size of the headers? Where is the key size + value size + header size calculated?
5. Jul 2018 14:15 by jiangtao....@zuora.com <mailto:jiangtao....@zuora.com>.INVALID: > sorry, just saw your email a little late. > > I am confused with why you said `I cannot increase the max size of messages > stored in Kafka` ? can we add some explanation ? > > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson < > jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>> > wrote: > >> >> What class is this in? What's the maximum header size? Is there >> documentation on this? >> >> 2. Jul 2018 18:29 by >> jiangtao....@zuora.com >> <mailto:jiangtao....@zuora.com>>> <>> mailto:jiangtao....@zuora.com >> <mailto:mailto:jiangtao....@zuora.com> >> >.INVALID: >> >> >> > You can consider increase `max.request.size` a little big (the default >> > value is `1048576`.), after checking Kafka client source code, they count >> > [`key size` + `value size` + `header size` + others] together, so it's >> > possible the calculated size is a little bigger than the default value. >> > >> > please check with > >> https://kafka.apache.org <https://kafka.apache.org/> >> documentation/#configuration <>> https://kafka.apache.org >> <https://kafka.apache.org/> >> documentation/#configuration>> . >> > >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> jerryrichard...@tutanota.com >> > <mailto:jerryrichard...@tutanota.com>>> <mailto: >> jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>>> >> > >> wrote: >> > >> >> Hi all, >> >> >> >> I get this error even when my records are smaller than the 1000012 byte >> >> limit: >> >> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request >> >> included a message larger than the max message size the server will >> accept. >> >> >> >> How do I ensure that my producer doesn't send records that are too >> large? >> >> >> >> Thanks in advance for any suggestions and help. >> >> >>
