Hi Arvid, Just as another user of Kafka, I suspect the claim has to do with the producer's ability to perform batching and compression across produced data, so it should be more efficient in some cases, especially with smaller volumes of data and larger linger.ms settings.
However, like you, I've found that a single KafkaProducer instance can sometimes become a bottleneck. For instance, threads can be blocked in certain cases when they are producing to the same partition. In our application, we found a producer per thread wasn't necessary, but adding a small pool of producers (e.g. 4 producers shared by 32 threads) helped to improve performance. It probably varies quite a bit on your use-case and configuration. I do wonder if the Kafka team would consider it a bug in the cases where a single producer isn't faster, or if they'd consider it a doc bug where the recommendation should be a bit more nuanced. Hope that helps, Meg -----Original Message----- From: Arvid Sundbom <arvid.sundbo...@gmail.com> Sent: Monday, April 24, 2023 9:01 AM To: users@kafka.apache.org Subject: Producer Application Utilizing Multiple Threads CAUTION: External Email : Be wary of clicking links or if this claims to be internal. Hi! I have a question about executing a Kafka producer application, utilizing multiple threads. In the documentation for Kafka producers ( https://kafka.apache.org/23/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html) it says that "...sharing a single producer instance across threads will generally be faster than having multiple instances.". I am mainly wondering if there is any data to verify this claim? The reason for this is that I have carried out rather extensive performance tests, with varying levels of computational load per message produced, using between 1 and 24 threads, and it seems that there is really no situation in which a single, shared producer achieves a higher performance than if each executing thread is assigned its own producer instance. Kind regards, Arvid
