Marisa,

I have kicked off the video series on performance optimization for the
Kafka setup.

I will be working on the various configurations for latency, throughput,
availability and durability.

https://youtu.be/aPlbG349cXg

The first ones will be on latency and throughput which is what you are
interested in and then I will work on the demos for availability and
durability later.

This will be done in KRaft and Legacy mode with sample datasets of 1000,
100000, and 1000000 messages end-to-end with variations in the number of
producers, consumers and partitions.

I am looking forward to this.

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/


On Thu, Jan 6, 2022 at 10:17 PM Marisa Queen <marisa.queen...@gmail.com>
wrote:

> Wow, that's awesome! I wasn't expecting that. I truly appreciate your help
> and professionalism.
>
> > Let me find some time soon and I will do a video on that scenario
> optimized primarily for low latency and throughput. I will also compare how
> this performs when adjusted for durability and high availability.
>
> Take your time! That will be tremendously helpful. I was going to try to do
> that myself, but I'm sure that you have better expertise to tune the knobs
> for a more realistic and professional benchmark.
>
> I'm curious to see the numbers. Perhaps you can even start with the
> simplest of all setups: 1 producer, 1 topic, 1 partition, 1 consumer. 10
> million messages flowing. What messages-per-second number do you get? Then
> move to 1 producer, 1 topic, 4 partitions, 1 consumer. Did it get better or
> worse with the addition of multiple partitions?
>
> Thanks again, Israel.
>
> Cheers,
>
> M. Queen
>
>
> On Thu, Jan 6, 2022 at 11:52 PM Israel Ekpo <israele...@gmail.com> wrote:
>
> > Thanks for your response Marisa.
> >
> > This has been a very interesting discussion and I appreciate it.
> >
> > It is a bit of a challenge in the sense that I wish I had a demo ready to
> > go with similar use case and expectations  to easily explain what I have
> > been trying to convey
> >
> > I am always ready for a challenge like this and to fix this I will like
> to
> > do a demo soon with the 10 million message scenario you
> > originally mentioned in your first message to track the time end to end
> > while capturing other metrics.
> >
> > Let me find some time soon and I will do a video on that scenario
> optimized
> > primarily for low latency and throughput. I will also compare how this
> > performs when adjusted for durability and high availability
> >
> > I wish I had this demo ready before now it would have clarified a lot of
> > what I have been trying to explain regarding tuning the knobs for
> latency,
> > throughput, high availability and durability. By achieving what you are
> > willing to pay for I was only suggesting that highly performant system
> can
> > be expensive at times and I apologize if the tone came out wrong
> >
> >  I am grateful that you brought this up and it will give the community
> > something to reference in the future of similar questions come up
> regarding
> > benchmarks
> >
> > Thanks for bringing up the question, please let us know if you have
> > additional questions and you can reach out with any further questions or
> > feedback you may have.
> >
> > Thanks again
> >
> > Sincerely
> > Israel Ekpo
> >
> > On Thu, Jan 6, 2022 at 9:18 PM Marisa Queen <marisa.queen...@gmail.com>
> > wrote:
> >
> > > Hi Israel,
> > >
> > > > You can achieve any performance benchmark you are willing to pay for.
> > >
> > > Thanks for your email. Allow me to respectfully disagree. I believe
> that
> > > some systems are better than others when it comes to performance. The
> > idea
> > > that I can just take a slow system, multiply by 1 million, and then I
> > have
> > > a super fast system, is at the least misleading. Assuming the same
> > hardware
> > > for everyone, some languages are faster than others. Some algorithms
> are
> > > faster than others. Some architectures are more efficient than others.
> > Some
> > > protocols are faster than others.
> > >
> > > Take a binary search vs a linear search for example. Binary search is
> of
> > > course much faster and more efficient than linear search (for large
> > lists),
> > > but according to your rationale this is not a problem. Just buy enough
> > > machines to do linear search in parallel and you can boast 1 million
> > > searches per second. What an amazing search system you are deploying!
> It
> > > can do 1 million searches per second, that's more than enough for any
> > > system.
> > >
> > > 7 TRILLION messages per day for Kafka/LinkedIn sounds amazing when just
> > > thrown on the table. Using your example, a transportation company can
> > > transport 5 packages per day using one of its bicycles. Is the
> > architecture
> > > of this company efficient? Fast? According to your rationale, it does
> not
> > > matter! The company needs only to buy 1 million bikes, and now it can
> > boast
> > > about delivering 5 million packages per day. You can say the company
> is a
> > > large corporation, but when it comes to efficiency it is more like a
> > > dinosaur. It has a high chance of being replaced by other more
> efficient
> > > companies in the future.
> > >
> > > To summarize, low latency is crucial for finance applications. You
> can't
> > > just say: "don't worry, it is proven and it can do 7 trillion messages
> > per
> > > day". That just won't do it. A ceiling benchmark number, for latency
> and
> > > throughput, is paramount for any system that wants to operate in that
> > > industry. The answer is not "as much as you are willing to pay for".
> > >
> > > Cheers,
> > >
> > > M. Queen
> > >
> > >
> > > On Thu, Jan 6, 2022 at 8:53 PM Israel Ekpo <israele...@gmail.com>
> wrote:
> > >
> > > > Marisa,
> > > >
> > > > I do not agree with your assessment. There are several factors that
> > could
> > > > influence your performance numbers even with localhost. Your project
> > > should
> > > > be configured based on your own needs.
> > > >
> > > > Your throughput could go up or lower depending on how you are
> > configured
> > > > based on what is important for your use case(s).
> > > >
> > > > If you have other apps running on the machine that would impact your
> > > > results. If you only have a 2 CPU, 4GB laptop, obviously you cannot
> > > compare
> > > > the results with a server that has 256GB of RAM and 64 Cores.
> > > >
> > > > Also, do not measure it in terms of messages per second but more in
> > terms
> > > > of data volume per second. A throughput of 100GBps will give you 100
> > > > messages per second 1 GB per message or 100,000 messages per second
> at
> > > 1KB
> > > > each if you have smaller messages the same volume will give a higher
> > > count
> > > > of messages for the same unit time.
> > > >
> > > > Take a look at the reference architecture and this best practices
> > > document
> > > > for how to optimize your performance based on your project goals
> > > > (durability, latency, throughput and availability)
> > > >
> > > > Confluent Platform Reference Architecture - Confluent
> > > > <
> > > >
> > >
> >
> https://www.confluent.io/thank-you/resources/apache-kafka-confluent-enterprise-reference-architecture/
> > > > >
> > > > Kafka Best Practices: Build, Monitor & Optimize Kafka in Confluent
> > Cloud
> > > > <
> > > >
> > >
> >
> https://www.confluent.io/thank-you/resources/recommendations-developers-using-confluent-cloud/
> > > > >
> > > >
> > > > Everybody's scenario and use case will impact how they set up their
> > > > project. You cannot look at another project and use their numbers for
> > > your
> > > > own set up. That is generally a bad idea and the better answer is
> that
> > > you
> > > > will need to define your project objectives and then figure out what
> is
> > > > needed to achieve those goals.
> > > >
> > > > The better question is to take a look at what volume throughput,
> > > retention
> > > > policy and period as well as environment and then figure out the
> > capacity
> > > > planning necessary to support what you need.
> > > >
> > > > You can achieve any performance benchmark you are willing to pay
> for. I
> > > am
> > > > not a fan of just blinding copying other peoples numbers and using it
> > out
> > > > of context in benchmarks comparisons.
> > > >
> > > > Take a look at the capacity planner and sizing calculator to figure
> out
> > > > what hardware and infrastructure you need for your scenario
> > > >
> > > > Sizing Calculator for Apache Kafka and Confluent Platform (
> > eventsizer.io
> > > )
> > > > <https://eventsizer.io/>
> > > >
> > > > I hope this is more useful.
> > > >
> > > >
> > > > Israel Ekpo
> > > > Lead Instructor, IzzyAcademy.com
> > > > https://www.youtube.com/c/izzyacademy
> > > > https://izzyacademy.com/
> > > >
> > > >
> > > > On Thu, Jan 6, 2022 at 6:07 PM Marisa Queen <
> marisa.queen...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Joris,
> > > > >
> > > > > Thank you so much, friend!
> > > > >
> > > > > > I appreciate that setting up everything on localhost will be
> easier
> > > and
> > > > > lead to big numbers, but bear in mind that it's typically all the
> > other
> > > > > real-life stuff (remote connections, replication, at-least once,
> ...)
> > > > that
> > > > > causes massive slowdowns compared to localhost
> > > > >
> > > > > Totally agree! But we must establish a ceiling first. If this
> > > > > super-good-loopback number doesn't look good, then one has no
> > business
> > > > > moving forward with Kafka to the more complex (and of course
> slower)
> > > > stuff.
> > > > >
> > > > > The purpose of the ceiling is that. It is your maximum ambition
> > > > represented
> > > > > by a number. You can't go any higher than that. At least with
> Kafka.
> > > > >
> > > > > Agree?
> > > > >
> > > > > Cheers,
> > > > >
> > > > > M. Queen
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 6, 2022 at 3:51 PM Joris Peeters <
> > > joris.mg.peet...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > These tutorials - though quite a bit outdated - seem quite
> useful:
> > > > > >
> > http://cloudurable.com/blog/kafka-tutorial-kafka-producer/index.html
> > > > > (and
> > > > > > the follow-ups).
> > > > > > Ends up being close to how I write this in Java, and tutorial 13
> > > talks
> > > > > > about batching and acks etc, which you'll need in order to tune
> to
> > > > > maximise
> > > > > > your throughput.
> > > > > >
> > > > > > I'm sure someone else has better example resources.
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 6, 2022 at 6:25 PM Marisa Queen <
> > > marisa.queen...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Joris,
> > > > > > >
> > > > > > > Thank you so much. I plan to write a Java Consumer and a Java
> > > > Producer,
> > > > > > for
> > > > > > > my benchmark. Do you recommend an example that I can use as a
> > > > reference
> > > > > > to
> > > > > > > write my basic Java producer and simple Java consumer? I'll for
> > > sure
> > > > > > share
> > > > > > > the through number I get with the community. Maybe even write a
> > > blog
> > > > > post
> > > > > > > about it. I hope it is more than 23 messages per second per
> > > partition
> > > > > > > :PPPPP
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > M. Queen
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 6, 2022 at 2:14 PM Joris Peeters <
> > > > > joris.mg.peet...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I'd just follow the instructions in
> > > > > > https://kafka.apache.org/quickstart
> > > > > > > to
> > > > > > > > set up Kafka and Zookeeper on a single node, by running the
> > Java
> > > > > > > processes
> > > > > > > > directly. Or can run in Docker.
> > > > > > > >
> > > > > > > > For the producer and consumer I'd personally use Python, as
> > it's
> > > > the
> > > > > > > > easiest to get going. You may want to look at
> > > > > > > > https://kafka-python.readthedocs.io/en/master/# (easier) and
> > > > > > > > https://github.com/confluentinc/confluent-kafka-python
> > (faster).
> > > > > > Similar
> > > > > > > > things exist for Go, Java, C++, ...
> > > > > > > > Or I'm sure there are some benchmark setups out there that
> you
> > > can
> > > > > > tweak
> > > > > > > a
> > > > > > > > little.
> > > > > > > >
> > > > > > > > I appreciate that setting up everything on localhost will be
> > > easier
> > > > > and
> > > > > > > > lead to big numbers, but bear in mind that it's typically all
> > the
> > > > > other
> > > > > > > > real-life stuff (remote connections, replication,
> > at-least-once,
> > > > ...)
> > > > > > > that
> > > > > > > > causes massive slowdowns compared to localhost, and those are
> > > > things
> > > > > > > banks
> > > > > > > > eventually tend to need (I work in finance industry myself).
> > What
> > > > > > you're
> > > > > > > > doing is a very useful benchmark, but I'd surround it with
> the
> > > > above
> > > > > > > > caveats to avoid overpromising.
> > > > > > > >
> > > > > > > > -J
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 6, 2022 at 4:58 PM Marisa Queen <
> > > > > marisa.queen...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Joris,
> > > > > > > > >
> > > > > > > > > I've spoken to him. His answers are below:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jan 6, 2022 at 1:37 PM Joris Peeters <
> > > > > > > joris.mg.peet...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > There's a few unknown parameters here that might
> influence
> > > the
> > > > > > > answer,
> > > > > > > > > > though. From the top of my head, at least
> > > > > > > > > > - How much replication of the data is needed (for high
> > > > > > availability),
> > > > > > > > and
> > > > > > > > > > how many acks for the producer? (If fire-and-forget it
> can
> > be
> > > > > > faster,
> > > > > > > > if
> > > > > > > > > > need to replicate and ack from 3 brokers in different
> DC's
> > > then
> > > > > > will
> > > > > > > be
> > > > > > > > > > slower)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Let's assume no high-availability for now, for simplicity's
> > > sake.
> > > > > > > > > Fire-and-forget like he said. We don't want to
> overcomplicate
> > > > this
> > > > > > > simple
> > > > > > > > > benchmark and we want the highest possible throughput
> number.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > - Transactions? (If end-to-end exactly-once then it's a
> lot
> > > > > slower)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Again no transactions. Let's keep it simple.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > - Size of the messages? (If each message is a GB it will
> > > > > obviously
> > > > > > be
> > > > > > > > > > slower)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > Let's assume 512 bytes. Powers of two are fun!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > - Distance and bandwidth between the producers, Kafka &
> the
> > > > > > > consumers?
> > > > > > > > > (If
> > > > > > > > > > the network links get saturated that would limit the
> > > > performance.
> > > > > > > > Latency
> > > > > > > > > > is likely less important than throughput, but if your
> > > consumers
> > > > > are
> > > > > > > in
> > > > > > > > > > Tokyo and the producer in London then it will likely also
> > be
> > > > > > slower)
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Loopback, same machine, for the love of God. Let's not even
> > go
> > > > > there.
> > > > > > > We
> > > > > > > > > want the highest possible throughput. I accept the limit of
> > the
> > > > > speed
> > > > > > > of
> > > > > > > > > light. If network particularities, and distances, are to be
> > > > > included
> > > > > > in
> > > > > > > > > this measurement then it is basically worth nothing.
> Loopback
> > > > > > > eliminates
> > > > > > > > > all those network variables that we surely don't want to
> > > include
> > > > in
> > > > > > the
> > > > > > > > > benchmark.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > FWIW, I find that the producer side is generally the
> > limiting
> > > > > > factor,
> > > > > > > > > > especially if there is only one.
> > > > > > > > > > I'd take a look at e.g. the Appendix test details on
> > > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://docs.confluent.io/2.0.0/clients/librdkafka/INTRODUCTION_8md.html
> > > > > > > > > .
> > > > > > > > > > I
> > > > > > > > > > haven't yet seen a faster Kafka impl than rdkafka, so
> those
> > > > would
> > > > > > be
> > > > > > > > > > reasonable upper bounds.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks for your reply, Joris. Can you point me to a Hello
> > World
> > > > > Kafka
> > > > > > > > > example, so I can set up this very basic and BARE BONES
> Kafka
> > > > > system,
> > > > > > > > > without any of the complications you correctly mentioned
> > > above? I
> > > > > > have
> > > > > > > 10
> > > > > > > > > million messages that I need to send from producers to
> > > > consumers. I
> > > > > > > have
> > > > > > > > 1
> > > > > > > > > topic, 1 producer for this topic, 4 partitions for this
> topic
> > > > and 4
> > > > > > > > > consumers, one for each partition. Everything loopback,
> same
> > > > > machine,
> > > > > > > no
> > > > > > > > > high-availability, transactions, etc. just KAFKA BARE
> BONES.
> > > What
> > > > > can
> > > > > > > be
> > > > > > > > > more trivial and basic than that?
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > M. Queen
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Jan 6, 2022 at 4:25 PM Marisa Queen <
> > > > > > > marisa.queen...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Israel,
> > > > > > > > > > >
> > > > > > > > > > > Your email is great, but I'm afraid to forward it to my
> > > > > customer
> > > > > > > > > because
> > > > > > > > > > it
> > > > > > > > > > > doesn't answer his question.
> > > > > > > > > > >
> > > > > > > > > > > I'm hoping that other members from this list will be
> able
> > > to
> > > > > give
> > > > > > > me
> > > > > > > > a
> > > > > > > > > > more
> > > > > > > > > > > NUMERIC answer, let's wait to see.
> > > > > > > > > > >
> > > > > > > > > > > Just to give you some follow up on your answer, when
> you
> > > say:
> > > > > > > > > > >
> > > > > > > > > > > > 30 passengers per driver or aircraft per day may not
> > > sound
> > > > > > > > impressive
> > > > > > > > > > but
> > > > > > > > > > > 750,000 passengers per day all together is how you
> should
> > > > look
> > > > > at
> > > > > > > it
> > > > > > > > > > >
> > > > > > > > > > > Well, with this rationality one can come up with any
> > > desired
> > > > > > > > throughput
> > > > > > > > > > > number by just adding more partitions. Do you see my
> > > customer
> > > > > > point
> > > > > > > > > that
> > > > > > > > > > > this does not make any sense? Adding more partitions
> also
> > > > does
> > > > > > not
> > > > > > > > come
> > > > > > > > > > for
> > > > > > > > > > > free, because messages need to be separated into the
> > newly
> > > > > > created
> > > > > > > > > > > partition and ordering will be lost. Order is important
> > for
> > > > > some
> > > > > > > > > > messages,
> > > > > > > > > > > so to keep adding more partitions towards an infinite
> > > > > throughput
> > > > > > is
> > > > > > > > not
> > > > > > > > > > an
> > > > > > > > > > > option.
> > > > > > > > > > >
> > > > > > > > > > > I've just spoken to him here, his reply was:
> > > > > > > > > > >
> > > > > > > > > > > "Marisa, I'm asking a very simple question for a very
> > basic
> > > > > Kafka
> > > > > > > > > > scenario.
> > > > > > > > > > > If I can't get an answer for that, then I'm in trouble.
> > Can
> > > > you
> > > > > > > > please
> > > > > > > > > > find
> > > > > > > > > > > out with your peers/community what is a good throughput
> > > > number
> > > > > to
> > > > > > > > have
> > > > > > > > > in
> > > > > > > > > > > mind for the scenario I've been describing. Again it
> is a
> > > > very
> > > > > > > basic
> > > > > > > > > and
> > > > > > > > > > > simple scenario: I have 10 million messages that I need
> > to
> > > > send
> > > > > > > from
> > > > > > > > > > > producers to consumers. Let's assume I have 1 topic, 1
> > > > producer
> > > > > > for
> > > > > > > > > this
> > > > > > > > > > > topic, 4 partitions for this topic and 4 consumers, one
> > for
> > > > > each
> > > > > > > > > > partition.
> > > > > > > > > > > What I would like to know is: How long is it going to
> > take
> > > > for
> > > > > > > these
> > > > > > > > 10
> > > > > > > > > > > million messages to travel all the way from the
> producer
> > to
> > > > the
> > > > > > > > > > consumers?
> > > > > > > > > > > That's the throughput performance number I'm interested
> > > in."
> > > > > > > > > > >
> > > > > > > > > > > I surely won't tell him: "Hey, that's easy, you have 4
> > > > > > partitions,
> > > > > > > > each
> > > > > > > > > > > partition according to LinkedIn can handle 23 messages
> > per
> > > > > > second,
> > > > > > > so
> > > > > > > > > we
> > > > > > > > > > > are looking for a 92 messages per second throughput
> > here!"
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > >
> > > > > > > > > > > M. Queen
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jan 6, 2022 at 12:58 PM Israel Ekpo <
> > > > > > israele...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Marisa
> > > > > > > > > > > >
> > > > > > > > > > > > I think there may be some confusion about the
> > throughput
> > > > for
> > > > > > each
> > > > > > > > > > > partition
> > > > > > > > > > > > and I want to explain briefly using some analogies
> > > > > > > > > > > >
> > > > > > > > > > > > Using transportation for example if we were to pick
> an
> > > > > airline
> > > > > > or
> > > > > > > > > > > > ridesharing organization to describe the volume of
> > > > customers
> > > > > > they
> > > > > > > > can
> > > > > > > > > > > > support per day we would have to look at how many
> total
> > > > > > customers
> > > > > > > > can
> > > > > > > > > > > > American Airlines service in a day or how many
> > customers
> > > > can
> > > > > > Uber
> > > > > > > > or
> > > > > > > > > > Lyft
> > > > > > > > > > > > serve in a day. We would not zero in on only the
> number
> > > of
> > > > > > > > customers
> > > > > > > > > a
> > > > > > > > > > > > particular driver can service or the number of
> > passengers
> > > > are
> > > > > > > > > > particular
> > > > > > > > > > > > aircraft than service in a day. That would be very
> > > limiting
> > > > > > > > > considering
> > > > > > > > > > > the
> > > > > > > > > > > > hundreds of thousands of aircrafts or drivers
> actively
> > > > > > > transporting
> > > > > > > > > > > > passengers in real time.
> > > > > > > > > > > >
> > > > > > > > > > > > 30 passengers per driver or aircraft per day may not
> > > sound
> > > > > > > > impressive
> > > > > > > > > > but
> > > > > > > > > > > > 750,000 passengers per day all together is how you
> > should
> > > > > look
> > > > > > at
> > > > > > > > it
> > > > > > > > > > > >
> > > > > > > > > > > > Partitions in Kafka are just a logical unit for
> > > organizing
> > > > > and
> > > > > > > > > storing
> > > > > > > > > > > data
> > > > > > > > > > > > within a Kafka topic. You should not base your
> analysis
> > > on
> > > > > just
> > > > > > > > what
> > > > > > > > > a
> > > > > > > > > > > > subunit of storage is able to support.
> > > > > > > > > > > >
> > > > > > > > > > > > I would recommend taking a look at Kafka Summit talks
> > on
> > > > > > > > performance
> > > > > > > > > > and
> > > > > > > > > > > > benchmarks to get some understanding how what Kafka
> is
> > > able
> > > > > to
> > > > > > do
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > applicable use cases in the Financial Services
> industry
> > > > > > > > > > > >
> > > > > > > > > > > > A lot of reputable organizations already trust Kafka
> > > today
> > > > > for
> > > > > > > > their
> > > > > > > > > > > needs
> > > > > > > > > > > > so this is already proven
> > > > > > > > > > > >
> > > > > > > > > > > > https://kafka.apache.org/powered-by
> > > > > > > > > > > >
> > > > > > > > > > > > I hope this helps.
> > > > > > > > > > > >
> > > > > > > > > > > > Israel Ekpo
> > > > > > > > > > > > Lead Instructor, IzzyAcademy.com
> > > > > > > > > > > > https://www.youtube.com/c/izzyacademy
> > > > > > > > > > > > https://izzyacademy.com/
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jan 6, 2022 at 10:01 AM Marisa Queen <
> > > > > > > > > > marisa.queen...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Cheers from NYC!
> > > > > > > > > > > > >
> > > > > > > > > > > > > I'm trying to give a performance number to a
> > potential
> > > > > client
> > > > > > > > (from
> > > > > > > > > > the
> > > > > > > > > > > > > financial market) who asked me the following
> > question:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *"If I have a Kafka system setup in the best way
> > > possible
> > > > > for
> > > > > > > > > > > > performance,
> > > > > > > > > > > > > what is an approximate number that I can have in
> mind
> > > for
> > > > > the
> > > > > > > > > > > throughput
> > > > > > > > > > > > of
> > > > > > > > > > > > > this system?"*
> > > > > > > > > > > > >
> > > > > > > > > > > > > The client proceeded to say:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *"What I want to know specifically, is how many
> > > messages
> > > > > per
> > > > > > > > second
> > > > > > > > > > > can I
> > > > > > > > > > > > > send from one side of my distributed system to the
> > > other
> > > > > side
> > > > > > > > with
> > > > > > > > > > > Apache
> > > > > > > > > > > > > Kafka."*
> > > > > > > > > > > > >
> > > > > > > > > > > > > And he concluded with:
> > > > > > > > > > > > >
> > > > > > > > > > > > > *"To give you an example, let's say I have 10
> million
> > > > > > messages
> > > > > > > > > that I
> > > > > > > > > > > > need
> > > > > > > > > > > > > to send from producers to consumers. Let's assume I
> > > have
> > > > 1
> > > > > > > > topic, 1
> > > > > > > > > > > > > producer for this topic, 4 partitions for this
> topic
> > > and
> > > > 4
> > > > > > > > > consumers,
> > > > > > > > > > > one
> > > > > > > > > > > > > for each partition. What I would like to know is:
> How
> > > > long
> > > > > is
> > > > > > > it
> > > > > > > > > > going
> > > > > > > > > > > to
> > > > > > > > > > > > > take for these 10 million messages to travel all
> the
> > > way
> > > > > from
> > > > > > > the
> > > > > > > > > > > > producer
> > > > > > > > > > > > > to the consumers? That's the throughput performance
> > > > number
> > > > > > I'm
> > > > > > > > > > > interested
> > > > > > > > > > > > > in."*
> > > > > > > > > > > > >
> > > > > > > > > > > > > I read in a reddit post yesterday (for some reason
> I
> > > > can't
> > > > > > find
> > > > > > > > the
> > > > > > > > > > > post
> > > > > > > > > > > > > anymore) that Kafka is able to handle 7 trillion
> > > messages
> > > > > per
> > > > > > > > day.
> > > > > > > > > > The
> > > > > > > > > > > > > LinkedIn article about it, says:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > *"We maintain over 100 Kafka clusters with more
> than
> > > > 4,000
> > > > > > > > brokers,
> > > > > > > > > > > which
> > > > > > > > > > > > > serve more than 100,000 topics and 7 million
> > > partitions.
> > > > > The
> > > > > > > > total
> > > > > > > > > > > number
> > > > > > > > > > > > > of messages handled by LinkedIn’s Kafka deployments
> > > > > recently
> > > > > > > > > > surpassed
> > > > > > > > > > > 7
> > > > > > > > > > > > > trillion per day."*
> > > > > > > > > > > > >
> > > > > > > > > > > > > The OP of the reddit post went on to say that
> > WhatsApp
> > > is
> > > > > > > > handling
> > > > > > > > > > > around
> > > > > > > > > > > > > 64 billion messages per day (740,000 msgs per sec x
> > 24
> > > x
> > > > > 60 x
> > > > > > > 60)
> > > > > > > > > and
> > > > > > > > > > > > that
> > > > > > > > > > > > > 7
> > > > > > > > > > > > > trillion for LinkedIn is a huge number, giving a
> > > whopping
> > > > > 81
> > > > > > > > > million
> > > > > > > > > > > > > messages per second for LinkedIn. But that doesn't
> > > matter
> > > > > for
> > > > > > > my
> > > > > > > > > > > > question.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 7 Trillion messages divided by 7 million partitions
> > > gives
> > > > > us
> > > > > > 1
> > > > > > > > > > million
> > > > > > > > > > > > > messages per day per partition. So to calculate the
> > > > > > throughput
> > > > > > > we
> > > > > > > > > do:
> > > > > > > > > > > > >
> > > > > > > > > > > > >     1 million divided by 60 divided by 60 divided
> by
> > 24
> > > > =>
> > > > > > *23
> > > > > > > > > > messages
> > > > > > > > > > > > per
> > > > > > > > > > > > > second per partition*
> > > > > > > > > > > > >
> > > > > > > > > > > > > We'll all agree that 23 messages per second per
> > > partition
> > > > > for
> > > > > > > > > > > throughput
> > > > > > > > > > > > > performance is very low, so I can't give this
> number
> > to
> > > > my
> > > > > > > > > potential
> > > > > > > > > > > > > client.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So my question is: *What number should I give to my
> > > > > potential
> > > > > > > > > > client?*
> > > > > > > > > > > > Note
> > > > > > > > > > > > > that he is a stubborn and strict bank CTO, so he
> > won't
> > > > take
> > > > > > any
> > > > > > > > > talk
> > > > > > > > > > > from
> > > > > > > > > > > > > me. He wants a mathematical answer using the
> > scientific
> > > > > > method.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Has anyone been in my shoes and can shed some light
> > on
> > > > this
> > > > > > > kafka
> > > > > > > > > > > > > throughput performance topic?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > >
> > > > > > > > > > > > > M. Queen
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > --
> > Israel Ekpo
> > Lead Instructor, IzzyAcademy.com
> > https://www.youtube.com/c/izzyacademy
> > https://izzyacademy.com/
> >
>

Reply via email to