Hi there One architecture pattern could be to have a number of app servers to receive the messages from the devices and send them to the Kafka cluster. This would let you handle authentication and authorisation from these devices, I don’t know your exact scenario but I don’t think you’d want 300K devices having direct write access to your Kafka cluster.
A second issue which would be more problematic is (as I understand it) every time a producer connects or disconnects to a cluster, it causes a rebalance of the producer -> partition mapping. Transient disconnections or restarts would cause some issues with that number of devices connecting and disconnecting. Daniel. On Tuesday, 24 June 2014 at 12:07 pm, 229hgcqi8+ag6x9...@guerrillamail.com wrote: > Hi, I’m new. We are considering using Kafka for our next generation product. > We have 300K devices that are connected to our cloud backend. We are > currently using a set of web services for our headless client devices to send > state information. We are moving to a message-based interface. > > Does Kafka support a use case of many thousands of client/producers? I can’t > seem to find anyone using Kafka in this way. It appears most user have the > small number of producers inside the data center. > > It also seems the producers need to know a lot of state information about the > Kafka cluster. Why should a client have to be involved selecting the right > partition for a given topic? > > Or maybe I am total confused on how to correctly design for Kafka. > > Any help / guidance is appreciated. > > > > > > ---- > Sent using GuerrillMail.com (http://GuerrillMail.com) > Block or report abuse: > https://www.guerrillamail.com/abuse/?a=VFJxFx4gSLUTgw%2F68W4ecRzCA8WC1Q%3D%3D > >
