Hello,

I have a proposal for an architecture on my desk, where people want
to store huge binary files (like images and videos up to a size of several
GB) 
in RiakCS. But the connection to RiakCS is supposed to work through Apache
Kafka,
so there will be a Kafka producer fetching the files from the source and
sending them to a Kafka-RiakCS consumer.
Now when I look into the Kafka configuration options
(http://kafka.apache.org/08/configuration.html)
I see that message.max.bytes is 1000000 by default, which would be much
too 
small for huge binary files like videos.
So my questions are:
Can this size be increased to support also messages with a size
of several GB? Has anyone already tried this? Are Kafka brokers,
consumers and producers able to handle such a message size?
Will setting such a huge limit on the message size have any impact
on the performance of transporting smaller messages?
Or should we better let our Kafka producers bypass Kafka, when
they encounter such huge binary files at the source and
let them store these files directly in RiakCS?

Best Regards,

Andreas Maier

Reply via email to