I agree with Tom but ... ... to reply Christina question I guess that Anas thought about this kind of solution in relation to the simplicity to read data from a specific devices from a backend service point of view.
Using one topic per device means that on the backend side you know exactly which is the topic to read to get data from device X. When you start using topic with more partitions and a key for device to determine the partition destination, on the backend side you have to read the entire topic (with data across different device) and get information about the key to understand which is the device sender. I'm only guessing that it could be the reason ... Btw ... +1 what Tom said Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor Twitter : @ppatierno Linkedin : paolopatierno Blog : DevExperience > Date: Mon, 16 May 2016 05:26:31 -0700 > Subject: Re: Increase number of topic in Kafka leads zookeeper fail > From: christian.po...@gmail.com > To: users@kafka.apache.org > > +1 what Tom said. > > Curious though Anas, what motivated you to try a topic per device? was > there something regarding management or security that you believe you can > achieve with topic per device? > > On Mon, May 16, 2016 at 4:11 AM, Tom Crayford <tcrayf...@heroku.com> wrote: > > > Hi there, > > > > Generally you don't use a single topic per device in this use case, but one > > topic with some number of partitions and the key distribution based on > > device id. Kafka isn't designed for millions of low volume topics, but a > > few high volume ones. > > > > Thanks > > > > Tom Crayford > > Heroku Kafka > > > > On Mon, May 16, 2016 at 5:23 AM, Anas A <anas.2...@gmail.com> wrote: > > > > > We plan to use kafka as a message broker for IoT use case, where each > > > device is considered as unique topic. when I simulated 10 message per > > > second to 10 thousand topics zookeeper is getting bottle neck,all Kafka > > > monitoring tools fails to read the throughput values and number of topics > > > from JMX port because of that. will tuning zookeeper will solve the > > issues. > > > where In IoT use case there will be millions of device polling data to > > > millions of topics. I want to make sure the approach is perfect to go. > > > Please suggest. > > > > > > > > > *Thanks & Regards,* > > > > > > > > > Anas A > > > DBA, Trinity Mobility > > > [image: facebook] <https://www.facebook.com/anas.24aj> [image: twitter] > > > <https://twitter.com/anas24aj> [image: linkedin] > > > <http://in.linkedin.com/in/anas24aj> [image: googleplus] > > > <https://plus.google.com/u/0/+anasA24aj/> > > > +917736368236 > > > anas.2...@gmail.com > > > Bangalore > > > > > > > > > -- > *Christian Posta* > twitter: @christianposta > http://www.christianposta.com/blog > http://fabric8.io
