Setting max.request.size does not prevent RecordTooLargeException from being thrown. Thanks for the header info.
9. Jul 2018 09:20 by jiangtao....@zuora.com <mailto:jiangtao....@zuora.com>.INVALID: > you can configure max.request.size > <> https://kafka.apache.org/documentation/#configuration > <https://kafka.apache.org/documentation/#configuration>> > when you create > Kafka producer, about the second questions, Where is the key size + value > size + header size calculated? you can check ` > > org.apache.kafka.common.record.AbstractRecords.estimateSizeInBytesUpperBound() > {...} > ` > > Ty > > > On Fri, Jul 6, 2018 at 12:42 AM, Jerry Richardson < > jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>> > wrote: > >> My administrator will not allow messages larger than 1MB to be stored in >> Kafka. >> How can I limit the size of my messages to 1MB? If I have a message larger >> than 1MB, I want to truncate or throw away the message to avoid the >> RecordTooLargeException. What is the max size of the headers? Where is >> the key size + value size + header size calculated? >> >> 5. Jul 2018 14:15 by >> jiangtao....@zuora.com >> <mailto:jiangtao....@zuora.com>>> <>> mailto:jiangtao....@zuora.com >> <mailto:mailto:jiangtao....@zuora.com> >> >.INVALID: >> >> >> > sorry, just saw your email a little late. >> > >> > I am confused with why you said `I cannot increase the max size of >> messages >> > stored in Kafka` ? can we add some explanation ? >> > >> > On Thu, Jul 5, 2018 at 10:09 AM, Jerry Richardson < >> > >> jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>>> >> > >> <>> mailto:jerryrichard...@tutanota.com >> > >> <mailto:mailto:jerryrichard...@tutanota.com>>> >> > >> wrote: >> > >> >> >> >> What class is this in? What's the maximum header size? Is there >> >> documentation on this? >> >> >> >> 2. Jul 2018 18:29 by >> >> jiangtao....@zuora.com >> >> <mailto:jiangtao....@zuora.com>>> <mailto: >> jiangtao....@zuora.com <mailto:jiangtao....@zuora.com>>> >>> <>> >> >> mailto:jiangtao....@zuora.com <mailto:mailto:jiangtao....@zuora.com>>> >> <mailto: >> mailto:jiangtao....@zuora.com <mailto:mailto:jiangtao....@zuora.com>>> > >> >> >.INVALID: >> >> >> >> >> >> > You can consider increase `max.request.size` a little big (the >> default >> >> > value is `1048576`.), after checking Kafka client source code, they >> count >> >> > [`key size` + `value size` + `header size` + others] together, so it's >> >> > possible the calculated size is a little bigger than the default >> value. >> >> > >> >> > please check with > >> >> https://kafka.apache.org >> >> > <https://kafka.apache.org>>> < >> https://kafka.apache.org <https://kafka.apache.org/>>> > >> >> documentation/#configuration <>> >> https://kafka.apache.org >> >> <https://kafka.apache.org>>> < >> https://kafka.apache.org <https://kafka.apache.org/>>> > >> >> documentation/#configuration>> . >> >> > >> >> > On Mon, Jul 2, 2018 at 5:08 PM, <> >> >> jerryrichard...@tutanota.com >> >> > <mailto:jerryrichard...@tutanota.com> >> <>> mailto:jerryrichard...@tutanota.com >> <mailto:mailto:jerryrichard...@tutanota.com>>> >>> <mailto: >> >> >> jerryrichard...@tutanota.com <mailto:jerryrichard...@tutanota.com>>> >> >> >> <>> mailto:jerryrichard...@tutanota.com >> >> >> <mailto:mailto:jerryrichard...@tutanota.com>>> >>> >> >> > wrote: >> >> > >> >> >> Hi all, >> >> >> >> >> >> I get this error even when my records are smaller than the 1000012 >> byte >> >> >> limit: >> >> >> >> >> >> org.apache.kafka.common.errors.RecordTooLargeException: The request >> >> >> included a message larger than the max message size the server will >> >> accept. >> >> >> >> >> >> How do I ensure that my producer doesn't send records that are too >> >> large? >> >> >> >> >> >> Thanks in advance for any suggestions and help. >> >> >> >> >> >>
