Hi Ola, I would suggest you can go with single Topic with multiple partitions. Once the data gets received from the Topic, you can do a DB update kind of a stuff to store the data , then use the data for analysing.
Also, the below URL can be used to do the Topic sizing. eventsizer.io Thanks C Suresh On Thursday, December 30, 2021, Ola Bissani <ola.biss...@easysoft.com.lb> wrote: > Dears, > > I'm looking for a way to get real-time updates using my service, I believe > kafka is the way to go but I still have an issue on how to use it. > > My system gets data from devices using GPRS, I then read this data and > analyze it to check what action I should do afterwards. I need the > analyzing step to be as fast as possible. I was thinking of two options: > > The first option is to gather all the data sent from all the devices into > one huge topic and then getting all the data from this topic and analyzing > it. The downside of this option is that the data analysis step is delaying > my work since I was to loop through the topic data, on the other hand the > advantage is that I have a manageable number of topics ( only 1 topic). > > The other option is to divide the data I'm gathering into several small > topics by allowing each device to have its own topic, take into > consideration that the number of devices is large, I'm talking about more > that 5000 devices. The downside of this option is that I have thousands of > topics, where the advantage is that each topic will have a manageable > amount of data allowing me to get my analysis done in much more reasonable > time. > > Can you advise on what option is better and whether there is a third > option that I'm not considering, > > > > Best Regards > Ola Bissani > Developer Manager > Easysoft > Mobile Lebanon : +961 3 61 16 90 > Office Lebanon :+961 1 33 55 15/17 > E mail: ola.biss...@easysoft.com.lb > web site:www.easysoft.com.lb > "Tailored to Perfection" > > > The information transmitted is intended only for the person or entity to > which it is addressed and it may contain proprietary, > business-confidential, and/or legally privileged information. If you are > not the intended recipient of this email you are hereby notified that any > use, review, retransmission, dissemination, distribution, reproduction or > any other action taken in reliance upon this email is strictly prohibited. > If you have received this email in error, please contact the sender and > delete this email and its contents from any computer. Any views expressed > in this email are those of the individual sender and may not necessarily > reflect the views of the company. > > Please consider the environmet before printing this email. > > > -----Original Message----- > From: Wes Peng <pent...@gmail.com> > Sent: Thursday, December 23, 2021 10:11 PM > To: users@kafka.apache.org > Subject: Re: Kafka Topics > > That depends on your resources such as ram, disk etc. General speaking > there is no problem. > > Regards > > > > Dears, > > > > I'm new to using Kafka, and I was wondering up to how many topics can > > Kafka Handle. I'm trying to use Kafka but using it I will be obliged > > to create thousands of topics to keep up with my data. Will Kafka on > > my server handle this kind of data? > > > > Thank you, > > > > > > Best Regards > > > > Ola Bissani > > > > Developer Manager > > > > Easysoft > > > > Mobile Lebanon : +961 3 61 16 90 > > > > Office Lebanon :+961 1 33 55 15/17 > > > > E mail: ola.biss...@easysoft.com.lb > > > > web site:www.easysoft.com.lb > > > > "Tailored to Perfection" > > > > > > > > > > The information transmitted is intended only for the person or entity > > to which it is addressed and it may contain proprietary, > > business-confidential, and/or legally privileged information. If you > > are not the intended recipient of this email you are hereby notified > > that any use, review, retransmission, dissemination, distribution, > > reproduction or any other action taken in reliance upon this email is > strictly prohibited. > > If you have received this email in error, please contact the sender > > and delete this email and its contents from any computer. Any views > > expressed in this email are those of the individual sender and may not > > necessarily reflect the views of the company. > > > > Please consider the environmet before printing this email. > > > > > >
