Hi all, As far as I understand this Jira https://issues.apache.org/jira/browse/KAFKA-4169, unfortunately max.request.size works on *uncompressed* messages. Which means producer won't be able to send messages larger than that limit, no matter compression. The fragment above is still right though, because it considers only broker side.
I'm thinking if it was possible (but somewhat risky) to define max.request.size on producer side larger (e.g. 2MB) to account the typical compression ratio of the data (knowing it apriori). Will that work? Thanks, Krzysztof 2017-06-22 15:55 GMT+02:00 mayank rathi <mayank.ra...@gmail.com>: > Hello Eli, > > This is from Kafka: Definitive Guide ( by Neha Narkhede , Gwen Shapira , > and Todd Palino) , Chapter 2. Installing Kafka > > "The Kafka broker limits the maximum size of a message that can be > produced, configured by the message.max.bytes parameter which defaults to > 1000000, or 1 megabyte. A producer which tries to send a message larger > than this will receive an error back from the broker and the message will > not be accepted. As with all byte sizes specified on the broker, this > configuration deals with compressed message size, which means that > producers can send messages that are much larger than this value > uncompressed, provided they compress down to under the configured > message.max.bytes size." > > Thanks!! > > On Thu, Jun 22, 2017 at 4:18 AM, Eli Jordan <elias.k.jor...@gmail.com> > wrote: > > > Thanks for the reply Mayank. Do you know if this is documented somewhere? > > I wasnt able to find mention of it. > > > > Thanks > > Eli > > > > > On 22 Jun 2017, at 05:50, mayank rathi <mayank.ra...@gmail.com> wrote: > > > > > > If you are compressing messages than size of "compressed" message > should > > be > > > less than what's specified in these parameters. > > > > > > On Sat, Jun 17, 2017 at 7:46 PM, Eli Jordan <elias.k.jor...@gmail.com> > > > wrote: > > > > > >> Hi > > >> > > >> max.message.bytes controls the maximum message size the kafka server > > will > > >> process > > >> > > >> message.max.bytes controls the maximum message size the consumer will > > >> process > > >> > > >> max.request.size controls the maximum request size for the producer > > >> > > >> Whats not clear to me (and I can't find documented anywhere) is if the > > >> message size limits are imposed on compressed or uncompressed > messages, > > >> when compression is enabled. > > >> > > >> Note: I'm use Kafka 0.10.2.0 if that makes any difference. > > >> > > >> Any pointers or advice on this would be greatly appreciated. > > >> > > >> Thanks > > >> Eli > > > > > > > > > > > > > > > -- > > > NOTICE: This email message is for the sole use of the intended > > recipient(s) > > > and may contain confidential and privileged information. Any > unauthorized > > > review, use, disclosure or distribution is prohibited. If you are not > the > > > intended recipient, please contact the sender by reply email and > destroy > > > all copies of the original message. > > > > > > -- > NOTICE: This email message is for the sole use of the intended recipient(s) > and may contain confidential and privileged information. Any unauthorized > review, use, disclosure or distribution is prohibited. If you are not the > intended recipient, please contact the sender by reply email and destroy > all copies of the original message. >
