When using Apache Kafka, the maximum message size can be defined in the configuration. If you have a lot of similarly sized messages, you can probably make a good estimate of a max message size. When the message sizes are highly variable, that's less certain.
This presentation from an engineer at LinkedIn: https://www.slideshare.net/JiangjieQin/handle-large-messages-in-apache-kafka-58692297 describes a Kafka message segmentation facility they created. I poked around looking for an implementation of this concept, but came up short. Before I go off an reimplement the wheel, does anyone know if there is something out there already? I'd be using it with Dana Powers' Kafka package: https://pypi.python.org/pypi/kafka/1.3.4.1 Thx, Skip -- https://mail.python.org/mailman/listinfo/python-list
