The message size limit is imposed to protect the brokers and consumers from running out of memory. The consumer does not have support for streaming a message and has to allocate memory to be able to read the largest message. You could try compressing the files but I'm not sure if that will get you as much space saving to make it feasible for Kafka usage.
Thanks, Neha On Sep 4, 2013 5:29 AM, "Maier, Dr. Andreas" <andreas.ma...@asideas.de> wrote: > Hello, > > I have a proposal for an architecture on my desk, where people want > to store huge binary files (like images and videos up to a size of several > GB) > in RiakCS. But the connection to RiakCS is supposed to work through Apache > Kafka, > so there will be a Kafka producer fetching the files from the source and > sending them to a Kafka-RiakCS consumer. > Now when I look into the Kafka configuration options > (http://kafka.apache.org/08/configuration.html) > I see that message.max.bytes is 1000000 by default, which would be much > too > small for huge binary files like videos. > So my questions are: > Can this size be increased to support also messages with a size > of several GB? Has anyone already tried this? Are Kafka brokers, > consumers and producers able to handle such a message size? > Will setting such a huge limit on the message size have any impact > on the performance of transporting smaller messages? > Or should we better let our Kafka producers bypass Kafka, when > they encounter such huge binary files at the source and > let them store these files directly in RiakCS? > > Best Regards, > > Andreas Maier > >
