I can confirm that the message size check in the producer works on the uncompressed size as of 0.11.0.1, as I had to investigate this internally. :)
I've got a similar problem with messages that can occasionally exceed this limit. We're taking the approach of enforcing a hard size limit when events are consumed at the start of our processing pipeline and setting a "soft limit" of higher than that as our max.request.size for all subsequent applications on that pipeline. I think you will need to make sure that your brokers are configured to replicate messages that large. I think if you push a single message that's 2MB, but the brokers have limits on their replication size you could run into issues? Could be wrong though. Working from a vague memory of broker configuration at the moment. On Wed, Oct 18, 2017 at 11:20 AM Krzysztof Zarzycki <k.zarzy...@gmail.com> wrote: > Hi all, > As far as I understand this Jira > https://issues.apache.org/jira/browse/KAFKA-4169, unfortunately > max.request.size works on *uncompressed* messages. Which means producer > won't be able to send messages larger than that limit, no matter > compression. The fragment above is still right though, because it considers > only broker side. > > I'm thinking if it was possible (but somewhat risky) to define > max.request.size on producer side larger (e.g. 2MB) to account the typical > compression ratio of the data (knowing it apriori). Will that work? > > > Thanks, > Krzysztof > > > 2017-06-22 15:55 GMT+02:00 mayank rathi <mayank.ra...@gmail.com>: > > > Hello Eli, > > > > This is from Kafka: Definitive Guide ( by Neha Narkhede , Gwen Shapira , > > and Todd Palino) , Chapter 2. Installing Kafka > > > > "The Kafka broker limits the maximum size of a message that can be > > produced, configured by the message.max.bytes parameter which defaults to > > 1000000, or 1 megabyte. A producer which tries to send a message larger > > than this will receive an error back from the broker and the message will > > not be accepted. As with all byte sizes specified on the broker, this > > configuration deals with compressed message size, which means that > > producers can send messages that are much larger than this value > > uncompressed, provided they compress down to under the configured > > message.max.bytes size." > > > > Thanks!! > > > > On Thu, Jun 22, 2017 at 4:18 AM, Eli Jordan <elias.k.jor...@gmail.com> > > wrote: > > > > > Thanks for the reply Mayank. Do you know if this is documented > somewhere? > > > I wasnt able to find mention of it. > > > > > > Thanks > > > Eli > > > > > > > On 22 Jun 2017, at 05:50, mayank rathi <mayank.ra...@gmail.com> > wrote: > > > > > > > > If you are compressing messages than size of "compressed" message > > should > > > be > > > > less than what's specified in these parameters. > > > > > > > > On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan < > elias.k.jor...@gmail.com> > > > > wrote: > > > > > > > >> Hi > > > >> > > > >> max.message.bytes controls the maximum message size the kafka server > > > will > > > >> process > > > >> > > > >> message.max.bytes controls the maximum message size the consumer > will > > > >> process > > > >> > > > >> max.request.size controls the maximum request size for the producer > > > >> > > > >> Whats not clear to me (and I can't find documented anywhere) is if > the > > > >> message size limits are imposed on compressed or uncompressed > > messages, > > > >> when compression is enabled. > > > >> > > > >> Note: I'm use Kafka 0.10.2.0 if that makes any difference. > > > >> > > > >> Any pointers or advice on this would be greatly appreciated. > > > >> > > > >> Thanks > > > >> Eli > > > > > > > > > > > > > > > > > > > > -- > > > > NOTICE: This email message is for the sole use of the intended > > > recipient(s) > > > > and may contain confidential and privileged information. Any > > unauthorized > > > > review, use, disclosure or distribution is prohibited. If you are not > > the > > > > intended recipient, please contact the sender by reply email and > > destroy > > > > all copies of the original message. > > > > > > > > > > > -- > > NOTICE: This email message is for the sole use of the intended > recipient(s) > > and may contain confidential and privileged information. Any unauthorized > > review, use, disclosure or distribution is prohibited. If you are not the > > intended recipient, please contact the sender by reply email and destroy > > all copies of the original message. > > >
