Hello, I have a proposal for an architecture on my desk, where people want to store huge binary files (like images and videos up to a size of several GB) in RiakCS. But the connection to RiakCS is supposed to work through Apache Kafka, so there will be a Kafka producer fetching the files from the source and sending them to a Kafka-RiakCS consumer. Now when I look into the Kafka configuration options (http://kafka.apache.org/08/configuration.html) I see that message.max.bytes is 1000000 by default, which would be much too small for huge binary files like videos. So my questions are: Can this size be increased to support also messages with a size of several GB? Has anyone already tried this? Are Kafka brokers, consumers and producers able to handle such a message size? Will setting such a huge limit on the message size have any impact on the performance of transporting smaller messages? Or should we better let our Kafka producers bypass Kafka, when they encounter such huge binary files at the source and let them store these files directly in RiakCS?
Best Regards, Andreas Maier
