On the best number of partitions per topic for kafka based application and the best number of consumers per topic?

Given a business application that resorts into a message queue solution like Kafka, what is the best number of partitions to select for a given topic? what influences such a decision? On the other hand, say we want to achieve a maximal throughput of message consumption but at minimal resource

Re: High latency for Kafka transactions

On 2020/11/13 09:09:00, John Coleman wrote: > Hi, > > We have very high latency ~1 second when we send a small number of small > messages to Kafka. The whole group of messages is less than 1k. We are > thinking better to just send the messages together as 1 message containing a > collectio

Re: On the best number of partitions per topic for kafka based application and the best number of consumers per topic?

This depends on the design of your application and how you are using Kafka. For instance, if you are using Kafka as a message queuing app, using consumers within a consumer group to load balance, then you should create the topics with as many partitions as the max number of consumers within the sa

Re: On the best number of partitions per topic for kafka based application and the best number of consumers per topic?

Thanks, My question is mostly about dynamic resource optimization, say I configured my application with 30 partitions and then I managed for 30 consumers (within a consumer group) to read/process the produced messages, but say at for instance at some unpeak load, I realized that a single consu