Marisa, you might consider engaging someone at Confluent, maybe they can give you some case studies or whitepapers from similar use-cases in the financial industry. (and yes, Kafka is used in the financial industry) . A client asking you to "prove that Kafka performs/scales" seems like an unusual request. Furthermore, setting up a localhost pub/sub demo on a single machine (your laptop?) is so far removed from a real-world scenario I can't imagine how any numbers derived from that would be useful. Is there a target latency or throughput volume that your client needs to hit? (are there any requirements at all??)
-alex On Thu, Jan 6, 2022 at 8:52 PM Israel Ekpo <israele...@gmail.com> wrote: > Thanks for your response Marisa. > > This has been a very interesting discussion and I appreciate it. > > It is a bit of a challenge in the sense that I wish I had a demo ready to > go with similar use case and expectations to easily explain what I have > been trying to convey > > I am always ready for a challenge like this and to fix this I will like to > do a demo soon with the 10 million message scenario you > originally mentioned in your first message to track the time end to end > while capturing other metrics. > > Let me find some time soon and I will do a video on that scenario optimized > primarily for low latency and throughput. I will also compare how this > performs when adjusted for durability and high availability > > I wish I had this demo ready before now it would have clarified a lot of > what I have been trying to explain regarding tuning the knobs for latency, > throughput, high availability and durability. By achieving what you are > willing to pay for I was only suggesting that highly performant system can > be expensive at times and I apologize if the tone came out wrong > > I am grateful that you brought this up and it will give the community > something to reference in the future of similar questions come up regarding > benchmarks > > Thanks for bringing up the question, please let us know if you have > additional questions and you can reach out with any further questions or > feedback you may have. > > Thanks again > > Sincerely > Israel Ekpo > > On Thu, Jan 6, 2022 at 9:18 PM Marisa Queen <marisa.queen...@gmail.com> > wrote: > > > Hi Israel, > > > > > You can achieve any performance benchmark you are willing to pay for. > > > > Thanks for your email. Allow me to respectfully disagree. I believe that > > some systems are better than others when it comes to performance. The > idea > > that I can just take a slow system, multiply by 1 million, and then I > have > > a super fast system, is at the least misleading. Assuming the same > hardware > > for everyone, some languages are faster than others. Some algorithms are > > faster than others. Some architectures are more efficient than others. > Some > > protocols are faster than others. > > > > Take a binary search vs a linear search for example. Binary search is of > > course much faster and more efficient than linear search (for large > lists), > > but according to your rationale this is not a problem. Just buy enough > > machines to do linear search in parallel and you can boast 1 million > > searches per second. What an amazing search system you are deploying! It > > can do 1 million searches per second, that's more than enough for any > > system. > > > > 7 TRILLION messages per day for Kafka/LinkedIn sounds amazing when just > > thrown on the table. Using your example, a transportation company can > > transport 5 packages per day using one of its bicycles. Is the > architecture > > of this company efficient? Fast? According to your rationale, it does not > > matter! The company needs only to buy 1 million bikes, and now it can > boast > > about delivering 5 million packages per day. You can say the company is a > > large corporation, but when it comes to efficiency it is more like a > > dinosaur. It has a high chance of being replaced by other more efficient > > companies in the future. > > > > To summarize, low latency is crucial for finance applications. You can't > > just say: "don't worry, it is proven and it can do 7 trillion messages > per > > day". That just won't do it. A ceiling benchmark number, for latency and > > throughput, is paramount for any system that wants to operate in that > > industry. The answer is not "as much as you are willing to pay for". > > > > Cheers, > > > > M. Queen > > > > > > On Thu, Jan 6, 2022 at 8:53 PM Israel Ekpo <israele...@gmail.com> wrote: > > > > > Marisa, > > > > > > I do not agree with your assessment. There are several factors that > could > > > influence your performance numbers even with localhost. Your project > > should > > > be configured based on your own needs. > > > > > > Your throughput could go up or lower depending on how you are > configured > > > based on what is important for your use case(s). > > > > > > If you have other apps running on the machine that would impact your > > > results. If you only have a 2 CPU, 4GB laptop, obviously you cannot > > compare > > > the results with a server that has 256GB of RAM and 64 Cores. > > > > > > Also, do not measure it in terms of messages per second but more in > terms > > > of data volume per second. A throughput of 100GBps will give you 100 > > > messages per second 1 GB per message or 100,000 messages per second at > > 1KB > > > each if you have smaller messages the same volume will give a higher > > count > > > of messages for the same unit time. > > > > > > Take a look at the reference architecture and this best practices > > document > > > for how to optimize your performance based on your project goals > > > (durability, latency, throughput and availability) > > > > > > Confluent Platform Reference Architecture - Confluent > > > < > > > > > > https://www.confluent.io/thank-you/resources/apache-kafka-confluent-enterprise-reference-architecture/ > > > > > > > Kafka Best Practices: Build, Monitor & Optimize Kafka in Confluent > Cloud > > > < > > > > > > https://www.confluent.io/thank-you/resources/recommendations-developers-using-confluent-cloud/ > > > > > > > > > > Everybody's scenario and use case will impact how they set up their > > > project. You cannot look at another project and use their numbers for > > your > > > own set up. That is generally a bad idea and the better answer is that > > you > > > will need to define your project objectives and then figure out what is > > > needed to achieve those goals. > > > > > > The better question is to take a look at what volume throughput, > > retention > > > policy and period as well as environment and then figure out the > capacity > > > planning necessary to support what you need. > > > > > > You can achieve any performance benchmark you are willing to pay for. I > > am > > > not a fan of just blinding copying other peoples numbers and using it > out > > > of context in benchmarks comparisons. > > > > > > Take a look at the capacity planner and sizing calculator to figure out > > > what hardware and infrastructure you need for your scenario > > > > > > Sizing Calculator for Apache Kafka and Confluent Platform ( > eventsizer.io > > ) > > > <https://eventsizer.io/> > > > > > > I hope this is more useful. > > > > > > > > > Israel Ekpo > > > Lead Instructor, IzzyAcademy.com > > > https://www.youtube.com/c/izzyacademy > > > https://izzyacademy.com/ > > > > > > > > > On Thu, Jan 6, 2022 at 6:07 PM Marisa Queen <marisa.queen...@gmail.com > > > > > wrote: > > > > > > > Hi Joris, > > > > > > > > Thank you so much, friend! > > > > > > > > > I appreciate that setting up everything on localhost will be easier > > and > > > > lead to big numbers, but bear in mind that it's typically all the > other > > > > real-life stuff (remote connections, replication, at-least once, ...) > > > that > > > > causes massive slowdowns compared to localhost > > > > > > > > Totally agree! But we must establish a ceiling first. If this > > > > super-good-loopback number doesn't look good, then one has no > business > > > > moving forward with Kafka to the more complex (and of course slower) > > > stuff. > > > > > > > > The purpose of the ceiling is that. It is your maximum ambition > > > represented > > > > by a number. You can't go any higher than that. At least with Kafka. > > > > > > > > Agree? > > > > > > > > Cheers, > > > > > > > > M. Queen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 3:51 PM Joris Peeters < > > joris.mg.peet...@gmail.com > > > > > > > > wrote: > > > > > > > > > These tutorials - though quite a bit outdated - seem quite useful: > > > > > > http://cloudurable.com/blog/kafka-tutorial-kafka-producer/index.html > > > > (and > > > > > the follow-ups). > > > > > Ends up being close to how I write this in Java, and tutorial 13 > > talks > > > > > about batching and acks etc, which you'll need in order to tune to > > > > maximise > > > > > your throughput. > > > > > > > > > > I'm sure someone else has better example resources. > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 6:25 PM Marisa Queen < > > marisa.queen...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Hi Joris, > > > > > > > > > > > > Thank you so much. I plan to write a Java Consumer and a Java > > > Producer, > > > > > for > > > > > > my benchmark. Do you recommend an example that I can use as a > > > reference > > > > > to > > > > > > write my basic Java producer and simple Java consumer? I'll for > > sure > > > > > share > > > > > > the through number I get with the community. Maybe even write a > > blog > > > > post > > > > > > about it. I hope it is more than 23 messages per second per > > partition > > > > > > :PPPPP > > > > > > > > > > > > Cheers, > > > > > > > > > > > > M. Queen > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 2:14 PM Joris Peeters < > > > > joris.mg.peet...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > I'd just follow the instructions in > > > > > https://kafka.apache.org/quickstart > > > > > > to > > > > > > > set up Kafka and Zookeeper on a single node, by running the > Java > > > > > > processes > > > > > > > directly. Or can run in Docker. > > > > > > > > > > > > > > For the producer and consumer I'd personally use Python, as > it's > > > the > > > > > > > easiest to get going. You may want to look at > > > > > > > https://kafka-python.readthedocs.io/en/master/# (easier) and > > > > > > > https://github.com/confluentinc/confluent-kafka-python > (faster). > > > > > Similar > > > > > > > things exist for Go, Java, C++, ... > > > > > > > Or I'm sure there are some benchmark setups out there that you > > can > > > > > tweak > > > > > > a > > > > > > > little. > > > > > > > > > > > > > > I appreciate that setting up everything on localhost will be > > easier > > > > and > > > > > > > lead to big numbers, but bear in mind that it's typically all > the > > > > other > > > > > > > real-life stuff (remote connections, replication, > at-least-once, > > > ...) > > > > > > that > > > > > > > causes massive slowdowns compared to localhost, and those are > > > things > > > > > > banks > > > > > > > eventually tend to need (I work in finance industry myself). > What > > > > > you're > > > > > > > doing is a very useful benchmark, but I'd surround it with the > > > above > > > > > > > caveats to avoid overpromising. > > > > > > > > > > > > > > -J > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 4:58 PM Marisa Queen < > > > > marisa.queen...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Joris, > > > > > > > > > > > > > > > > I've spoken to him. His answers are below: > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 1:37 PM Joris Peeters < > > > > > > joris.mg.peet...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > There's a few unknown parameters here that might influence > > the > > > > > > answer, > > > > > > > > > though. From the top of my head, at least > > > > > > > > > - How much replication of the data is needed (for high > > > > > availability), > > > > > > > and > > > > > > > > > how many acks for the producer? (If fire-and-forget it can > be > > > > > faster, > > > > > > > if > > > > > > > > > need to replicate and ack from 3 brokers in different DC's > > then > > > > > will > > > > > > be > > > > > > > > > slower) > > > > > > > > > > > > > > > > > > > > > > > > > Let's assume no high-availability for now, for simplicity's > > sake. > > > > > > > > Fire-and-forget like he said. We don't want to overcomplicate > > > this > > > > > > simple > > > > > > > > benchmark and we want the highest possible throughput number. > > > > > > > > > > > > > > > > > > > > > > > > > - Transactions? (If end-to-end exactly-once then it's a lot > > > > slower) > > > > > > > > > > > > > > > > > > > > > > > > > Again no transactions. Let's keep it simple. > > > > > > > > > > > > > > > > > > > > > > > > > - Size of the messages? (If each message is a GB it will > > > > obviously > > > > > be > > > > > > > > > slower) > > > > > > > > > > > > > > > > > > > > > > > > > Let's assume 512 bytes. Powers of two are fun! > > > > > > > > > > > > > > > > > > > > > > > > > - Distance and bandwidth between the producers, Kafka & the > > > > > > consumers? > > > > > > > > (If > > > > > > > > > the network links get saturated that would limit the > > > performance. > > > > > > > Latency > > > > > > > > > is likely less important than throughput, but if your > > consumers > > > > are > > > > > > in > > > > > > > > > Tokyo and the producer in London then it will likely also > be > > > > > slower) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Loopback, same machine, for the love of God. Let's not even > go > > > > there. > > > > > > We > > > > > > > > want the highest possible throughput. I accept the limit of > the > > > > speed > > > > > > of > > > > > > > > light. If network particularities, and distances, are to be > > > > included > > > > > in > > > > > > > > this measurement then it is basically worth nothing. Loopback > > > > > > eliminates > > > > > > > > all those network variables that we surely don't want to > > include > > > in > > > > > the > > > > > > > > benchmark. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > FWIW, I find that the producer side is generally the > limiting > > > > > factor, > > > > > > > > > especially if there is only one. > > > > > > > > > I'd take a look at e.g. the Appendix test details on > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.confluent.io/2.0.0/clients/librdkafka/INTRODUCTION_8md.html > > > > > > > > . > > > > > > > > > I > > > > > > > > > haven't yet seen a faster Kafka impl than rdkafka, so those > > > would > > > > > be > > > > > > > > > reasonable upper bounds. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for your reply, Joris. Can you point me to a Hello > World > > > > Kafka > > > > > > > > example, so I can set up this very basic and BARE BONES Kafka > > > > system, > > > > > > > > without any of the complications you correctly mentioned > > above? I > > > > > have > > > > > > 10 > > > > > > > > million messages that I need to send from producers to > > > consumers. I > > > > > > have > > > > > > > 1 > > > > > > > > topic, 1 producer for this topic, 4 partitions for this topic > > > and 4 > > > > > > > > consumers, one for each partition. Everything loopback, same > > > > machine, > > > > > > no > > > > > > > > high-availability, transactions, etc. just KAFKA BARE BONES. > > What > > > > can > > > > > > be > > > > > > > > more trivial and basic than that? > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > M. Queen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 4:25 PM Marisa Queen < > > > > > > marisa.queen...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi Israel, > > > > > > > > > > > > > > > > > > > > Your email is great, but I'm afraid to forward it to my > > > > customer > > > > > > > > because > > > > > > > > > it > > > > > > > > > > doesn't answer his question. > > > > > > > > > > > > > > > > > > > > I'm hoping that other members from this list will be able > > to > > > > give > > > > > > me > > > > > > > a > > > > > > > > > more > > > > > > > > > > NUMERIC answer, let's wait to see. > > > > > > > > > > > > > > > > > > > > Just to give you some follow up on your answer, when you > > say: > > > > > > > > > > > > > > > > > > > > > 30 passengers per driver or aircraft per day may not > > sound > > > > > > > impressive > > > > > > > > > but > > > > > > > > > > 750,000 passengers per day all together is how you should > > > look > > > > at > > > > > > it > > > > > > > > > > > > > > > > > > > > Well, with this rationality one can come up with any > > desired > > > > > > > throughput > > > > > > > > > > number by just adding more partitions. Do you see my > > customer > > > > > point > > > > > > > > that > > > > > > > > > > this does not make any sense? Adding more partitions also > > > does > > > > > not > > > > > > > come > > > > > > > > > for > > > > > > > > > > free, because messages need to be separated into the > newly > > > > > created > > > > > > > > > > partition and ordering will be lost. Order is important > for > > > > some > > > > > > > > > messages, > > > > > > > > > > so to keep adding more partitions towards an infinite > > > > throughput > > > > > is > > > > > > > not > > > > > > > > > an > > > > > > > > > > option. > > > > > > > > > > > > > > > > > > > > I've just spoken to him here, his reply was: > > > > > > > > > > > > > > > > > > > > "Marisa, I'm asking a very simple question for a very > basic > > > > Kafka > > > > > > > > > scenario. > > > > > > > > > > If I can't get an answer for that, then I'm in trouble. > Can > > > you > > > > > > > please > > > > > > > > > find > > > > > > > > > > out with your peers/community what is a good throughput > > > number > > > > to > > > > > > > have > > > > > > > > in > > > > > > > > > > mind for the scenario I've been describing. Again it is a > > > very > > > > > > basic > > > > > > > > and > > > > > > > > > > simple scenario: I have 10 million messages that I need > to > > > send > > > > > > from > > > > > > > > > > producers to consumers. Let's assume I have 1 topic, 1 > > > producer > > > > > for > > > > > > > > this > > > > > > > > > > topic, 4 partitions for this topic and 4 consumers, one > for > > > > each > > > > > > > > > partition. > > > > > > > > > > What I would like to know is: How long is it going to > take > > > for > > > > > > these > > > > > > > 10 > > > > > > > > > > million messages to travel all the way from the producer > to > > > the > > > > > > > > > consumers? > > > > > > > > > > That's the throughput performance number I'm interested > > in." > > > > > > > > > > > > > > > > > > > > I surely won't tell him: "Hey, that's easy, you have 4 > > > > > partitions, > > > > > > > each > > > > > > > > > > partition according to LinkedIn can handle 23 messages > per > > > > > second, > > > > > > so > > > > > > > > we > > > > > > > > > > are looking for a 92 messages per second throughput > here!" > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > M. Queen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 12:58 PM Israel Ekpo < > > > > > israele...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Marisa > > > > > > > > > > > > > > > > > > > > > > I think there may be some confusion about the > throughput > > > for > > > > > each > > > > > > > > > > partition > > > > > > > > > > > and I want to explain briefly using some analogies > > > > > > > > > > > > > > > > > > > > > > Using transportation for example if we were to pick an > > > > airline > > > > > or > > > > > > > > > > > ridesharing organization to describe the volume of > > > customers > > > > > they > > > > > > > can > > > > > > > > > > > support per day we would have to look at how many total > > > > > customers > > > > > > > can > > > > > > > > > > > American Airlines service in a day or how many > customers > > > can > > > > > Uber > > > > > > > or > > > > > > > > > Lyft > > > > > > > > > > > serve in a day. We would not zero in on only the number > > of > > > > > > > customers > > > > > > > > a > > > > > > > > > > > particular driver can service or the number of > passengers > > > are > > > > > > > > > particular > > > > > > > > > > > aircraft than service in a day. That would be very > > limiting > > > > > > > > considering > > > > > > > > > > the > > > > > > > > > > > hundreds of thousands of aircrafts or drivers actively > > > > > > transporting > > > > > > > > > > > passengers in real time. > > > > > > > > > > > > > > > > > > > > > > 30 passengers per driver or aircraft per day may not > > sound > > > > > > > impressive > > > > > > > > > but > > > > > > > > > > > 750,000 passengers per day all together is how you > should > > > > look > > > > > at > > > > > > > it > > > > > > > > > > > > > > > > > > > > > > Partitions in Kafka are just a logical unit for > > organizing > > > > and > > > > > > > > storing > > > > > > > > > > data > > > > > > > > > > > within a Kafka topic. You should not base your analysis > > on > > > > just > > > > > > > what > > > > > > > > a > > > > > > > > > > > subunit of storage is able to support. > > > > > > > > > > > > > > > > > > > > > > I would recommend taking a look at Kafka Summit talks > on > > > > > > > performance > > > > > > > > > and > > > > > > > > > > > benchmarks to get some understanding how what Kafka is > > able > > > > to > > > > > do > > > > > > > and > > > > > > > > > the > > > > > > > > > > > applicable use cases in the Financial Services industry > > > > > > > > > > > > > > > > > > > > > > A lot of reputable organizations already trust Kafka > > today > > > > for > > > > > > > their > > > > > > > > > > needs > > > > > > > > > > > so this is already proven > > > > > > > > > > > > > > > > > > > > > > https://kafka.apache.org/powered-by > > > > > > > > > > > > > > > > > > > > > > I hope this helps. > > > > > > > > > > > > > > > > > > > > > > Israel Ekpo > > > > > > > > > > > Lead Instructor, IzzyAcademy.com > > > > > > > > > > > https://www.youtube.com/c/izzyacademy > > > > > > > > > > > https://izzyacademy.com/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 6, 2022 at 10:01 AM Marisa Queen < > > > > > > > > > marisa.queen...@gmail.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Cheers from NYC! > > > > > > > > > > > > > > > > > > > > > > > > I'm trying to give a performance number to a > potential > > > > client > > > > > > > (from > > > > > > > > > the > > > > > > > > > > > > financial market) who asked me the following > question: > > > > > > > > > > > > > > > > > > > > > > > > *"If I have a Kafka system setup in the best way > > possible > > > > for > > > > > > > > > > > performance, > > > > > > > > > > > > what is an approximate number that I can have in mind > > for > > > > the > > > > > > > > > > throughput > > > > > > > > > > > of > > > > > > > > > > > > this system?"* > > > > > > > > > > > > > > > > > > > > > > > > The client proceeded to say: > > > > > > > > > > > > > > > > > > > > > > > > *"What I want to know specifically, is how many > > messages > > > > per > > > > > > > second > > > > > > > > > > can I > > > > > > > > > > > > send from one side of my distributed system to the > > other > > > > side > > > > > > > with > > > > > > > > > > Apache > > > > > > > > > > > > Kafka."* > > > > > > > > > > > > > > > > > > > > > > > > And he concluded with: > > > > > > > > > > > > > > > > > > > > > > > > *"To give you an example, let's say I have 10 million > > > > > messages > > > > > > > > that I > > > > > > > > > > > need > > > > > > > > > > > > to send from producers to consumers. Let's assume I > > have > > > 1 > > > > > > > topic, 1 > > > > > > > > > > > > producer for this topic, 4 partitions for this topic > > and > > > 4 > > > > > > > > consumers, > > > > > > > > > > one > > > > > > > > > > > > for each partition. What I would like to know is: How > > > long > > > > is > > > > > > it > > > > > > > > > going > > > > > > > > > > to > > > > > > > > > > > > take for these 10 million messages to travel all the > > way > > > > from > > > > > > the > > > > > > > > > > > producer > > > > > > > > > > > > to the consumers? That's the throughput performance > > > number > > > > > I'm > > > > > > > > > > interested > > > > > > > > > > > > in."* > > > > > > > > > > > > > > > > > > > > > > > > I read in a reddit post yesterday (for some reason I > > > can't > > > > > find > > > > > > > the > > > > > > > > > > post > > > > > > > > > > > > anymore) that Kafka is able to handle 7 trillion > > messages > > > > per > > > > > > > day. > > > > > > > > > The > > > > > > > > > > > > LinkedIn article about it, says: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *"We maintain over 100 Kafka clusters with more than > > > 4,000 > > > > > > > brokers, > > > > > > > > > > which > > > > > > > > > > > > serve more than 100,000 topics and 7 million > > partitions. > > > > The > > > > > > > total > > > > > > > > > > number > > > > > > > > > > > > of messages handled by LinkedIn’s Kafka deployments > > > > recently > > > > > > > > > surpassed > > > > > > > > > > 7 > > > > > > > > > > > > trillion per day."* > > > > > > > > > > > > > > > > > > > > > > > > The OP of the reddit post went on to say that > WhatsApp > > is > > > > > > > handling > > > > > > > > > > around > > > > > > > > > > > > 64 billion messages per day (740,000 msgs per sec x > 24 > > x > > > > 60 x > > > > > > 60) > > > > > > > > and > > > > > > > > > > > that > > > > > > > > > > > > 7 > > > > > > > > > > > > trillion for LinkedIn is a huge number, giving a > > whopping > > > > 81 > > > > > > > > million > > > > > > > > > > > > messages per second for LinkedIn. But that doesn't > > matter > > > > for > > > > > > my > > > > > > > > > > > question. > > > > > > > > > > > > > > > > > > > > > > > > 7 Trillion messages divided by 7 million partitions > > gives > > > > us > > > > > 1 > > > > > > > > > million > > > > > > > > > > > > messages per day per partition. So to calculate the > > > > > throughput > > > > > > we > > > > > > > > do: > > > > > > > > > > > > > > > > > > > > > > > > 1 million divided by 60 divided by 60 divided by > 24 > > > => > > > > > *23 > > > > > > > > > messages > > > > > > > > > > > per > > > > > > > > > > > > second per partition* > > > > > > > > > > > > > > > > > > > > > > > > We'll all agree that 23 messages per second per > > partition > > > > for > > > > > > > > > > throughput > > > > > > > > > > > > performance is very low, so I can't give this number > to > > > my > > > > > > > > potential > > > > > > > > > > > > client. > > > > > > > > > > > > > > > > > > > > > > > > So my question is: *What number should I give to my > > > > potential > > > > > > > > > client?* > > > > > > > > > > > Note > > > > > > > > > > > > that he is a stubborn and strict bank CTO, so he > won't > > > take > > > > > any > > > > > > > > talk > > > > > > > > > > from > > > > > > > > > > > > me. He wants a mathematical answer using the > scientific > > > > > method. > > > > > > > > > > > > > > > > > > > > > > > > Has anyone been in my shoes and can shed some light > on > > > this > > > > > > kafka > > > > > > > > > > > > throughput performance topic? > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > > > > > > > > > > > > > M. Queen > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Israel Ekpo > Lead Instructor, IzzyAcademy.com > https://www.youtube.com/c/izzyacademy > https://izzyacademy.com/ >
