I was reading a nice summary article by Jun Rao on the implications of # of topics/partitions. http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
There are many trade-offs to be considered, as it looks. Finding the partition for a key: Can 'custom partitioner' be employed so that a consumer can derive(compute) a partition id from a key and access it directly instead of scanning all the partitions in a topic? A thought: Using a partition in place of a topic per device could, optionally, provide one level of hierarchy with some manageability options in that a topic can be for a certain type of IOT or certain geography of IOT, or ... etc. Thanks, Abhaya On Mon, May 16, 2016 at 5:30 AM, Paolo Patierno <ppatie...@live.com> wrote: > I agree with Tom but ... > > ... to reply Christina question I guess that Anas thought about this kind > of solution in relation to the simplicity to read data from a specific > devices from a backend service point of view. > > Using one topic per device means that on the backend side you know exactly > which is the topic to read to get data from device X. > When you start using topic with more partitions and a key for device to > determine the partition destination, on the backend side you have to read > the entire topic (with data across different device) and get information > about the key to understand which is the device sender. > > I'm only guessing that it could be the reason ... > > Btw ... +1 what Tom said > > Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor > Twitter : @ppatierno > Linkedin : paolopatierno > Blog : DevExperience > > > Date: Mon, 16 May 2016 05:26:31 -0700 > > Subject: Re: Increase number of topic in Kafka leads zookeeper fail > > From: christian.po...@gmail.com > > To: users@kafka.apache.org > > > > +1 what Tom said. > > > > Curious though Anas, what motivated you to try a topic per device? was > > there something regarding management or security that you believe you can > > achieve with topic per device? > > > > On Mon, May 16, 2016 at 4:11 AM, Tom Crayford <tcrayf...@heroku.com> > wrote: > > > > > Hi there, > > > > > > Generally you don't use a single topic per device in this use case, > but one > > > topic with some number of partitions and the key distribution based on > > > device id. Kafka isn't designed for millions of low volume topics, but > a > > > few high volume ones. > > > > > > Thanks > > > > > > Tom Crayford > > > Heroku Kafka > > > > > > On Mon, May 16, 2016 at 5:23 AM, Anas A <anas.2...@gmail.com> wrote: > > > > > > > We plan to use kafka as a message broker for IoT use case, where each > > > > device is considered as unique topic. when I simulated 10 message per > > > > second to 10 thousand topics zookeeper is getting bottle neck,all > Kafka > > > > monitoring tools fails to read the throughput values and number of > topics > > > > from JMX port because of that. will tuning zookeeper will solve the > > > issues. > > > > where In IoT use case there will be millions of device polling data > to > > > > millions of topics. I want to make sure the approach is perfect to > go. > > > > Please suggest. > > > > > > > > > > > > *Thanks & Regards,* > > > > > > > > > > > > Anas A > > > > DBA, Trinity Mobility > > > > [image: facebook] <https://www.facebook.com/anas.24aj> [image: > twitter] > > > > <https://twitter.com/anas24aj> [image: linkedin] > > > > <http://in.linkedin.com/in/anas24aj> [image: googleplus] > > > > <https://plus.google.com/u/0/+anasA24aj/> > > > > +917736368236 > > > > anas.2...@gmail.com > > > > Bangalore > > > > > > > > > > > > > > > -- > > *Christian Posta* > > twitter: @christianposta > > http://www.christianposta.com/blog > > http://fabric8.io > >
