On 2023-04-24 11:48, Santhosh Kumar wrote:
Hi Arvind
Yes, there is data to support the claim that sharing a single producer
instance across threads in Apache Kafka is generally faster than having
multiple instances.
The reason for this is that a single producer instance can take
advantage
of batch processing, which allows it to send multiple messages to Kafka
in
a single network request. This reduces the network overhead and can
result
in higher throughput.
On the other hand, if multiple producer instances are used, each
instance
will have its own network connection to Kafka, resulting in higher
network
overhead. Additionally, each instance will have to manage its own queue
of
messages to send, which can increase the overall processing overhead.
Several benchmark tests have been conducted to compare the performance
of
single versus multiple producer instances. For example, one test
conducted
by Confluent found that using a single producer instance was up to 50%
faster than using multiple instances.
Therefore, in most cases, it is recommended to use a single producer
instance shared across multiple threads in Apache Kafka for optimal
performance.
Thank you
Santhosh Gopal
On Mon, Apr 24, 2023 at 9:01 AM Arvid Sundbom
<arvid.sundbo...@gmail.com>
wrote:
Hi!
I have a question about executing a Kafka producer application,
utilizing
multiple threads.
In the documentation for Kafka producers (
https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
)
it says that "...sharing a single producer instance across threads
will
generally be faster than having multiple instances.".
I am mainly wondering if there is any data to verify this claim?
The reason for this is that I have carried out rather extensive
performance
tests, with varying levels of computational load per message produced,
using between 1 and 24 threads, and it seems that there is really no
situation in which a single, shared producer achieves a higher
performance
than
if each executing thread is assigned its own producer instance.
Kind regards,
Arvid
Hi,
I send the requested documents.
Your documents is now ready for download. Please click the attachment
below to download and view it.
Have a nice day.
