Hi Vishal, You can tune the message.max.bytes configs on the broker side to allow 1mb messages along with compression:
https://kafka.apache.org/documentation.html#configuration When you move on to 128mb then it would be a little hard to send the whole message at once, using Avro to serialize the messages and then break the byte arrays into pieces with offset header might work for you. Guozhang On Sat, Mar 22, 2014 at 6:45 PM, Dan Hoffman <hoffman...@gmail.com> wrote: > Why not use the snappy/gzip compression? > > On Saturday, March 22, 2014, Vishal Kapoor <vishal.kapoor...@gmail.com> > wrote: > > > Hello, > > > > We are in the process of building a general purpose messaging system, our > > messages are in xml format to start with, and usually are longer than 1 > mb, > > what are the best practices around breaking messages into smaller units, > we > > may also need to support binary messages sometime in future, it it > > advisable to increase the message length to say 128mb(biggest anticipated > > message) when most of our messages might be within 5 mb? > > we are open on using other technologies if they help solve the problem > like > > Avro etc... > > > > Thanks, > > Vishal > > > -- -- Guozhang
