Hi, I'm not entirely certain this is the right place to ask, but I have questions about the functioning of Apache Kafka to implement a publish/subscribe messaging system. I am investigating Kafka to see if it fits the needs of the company I work for, and I have quite a few questions I can't find any answer to either in the official doc or on the internet :
On the type and size of Kafka messages, I read the doc on RecordBatch and Records, but I couldn't find anything about the type of data expected in the Record value. From what I read on different websites and tutorials it seems that data can be anything as long as it can be converted into byte arrays. Moreover, I read that the default size of a message is 1MB, but as it is possible to configure it, what would be the maximum size of a message? - Is it possible to send files? Basically what we would want to do is streaming pictures, right now the system in use does not allow to send files so they are sent as binary data and assembled later. From what I understand of Kafka, files can be sent in a binary way, split and to be reassembled after. - Is it possible to send binary frames? Right now we have a few "custom" binary frame and we would like to know if this would still be possible when using Kafka. - Is it possible to send simple messages, such as a string + an int or a string + a string...? - What is the max size of a queue, if there is one? We would like to know what is the max volume of data that can be sent instantaneously (depending on the size of the queue, I suppose). On publish-subscribe more specifically : - Is it possible for a consumer to subscribe to a group of publishers? Otherwise I guess the easiest way to do so is to have all publishers from that group publish their messages in a specific topic. - Is it possible to subscribe to a group of topics? For example if we have several topics starting by "data", like data_1, data_2 etc, would it be possible to have consumers subscribe to "data*" ? - Is it possible to have sub-topics, and even sub-sub-sopics? From what I understand so far it doesn't seem possible, as topics are divided into partitions for replication. I know this is a lot of questions and that they are not really related. I hope I made myself clear enough and I thank you in advance for your answers. Regards, Héloïse Chevalier
