Thanks, My question is mostly about dynamic resource optimization, say I configured my application with 30 partitions and then I managed for 30 consumers (within a consumer group) to read/process the produced messages, but say at for instance at some unpeak load, I realized that a single consumer (within the same consumer group) could do consume all messages produced (production rate is low)? are you aware of any Kafka solution that would automatically join/leave consumers from within a consumer group dynamically e.g., based on how well the consumer group is doing to keep up with the production rate?
On 2020/11/14 22:36:34, Vinicius Scheidegger <vinicius.scheideg...@gmail.com> wrote: > This depends on the design of your application and how you are using Kafka. > > For instance, if you are using Kafka as a message queuing app, using > consumers within a consumer group to load balance, then you should create > the topics with as many partitions as the max number of consumers within > the same consumer group reading from it. > > For instance, in a regular load you believe you will have 8 consumers > reading from a topic, but at peak you believe you could have around 30 > consumers. You should then have at least 30 partitions - each consumer in a > consumer group is assigned to a partition. Having more consumers in a > single consumer group than partitions are not effective as they won't be > receiving any messages. > > But as I mentioned earlier different designs may lead to different > decisions. Try to understand first how partitions work. How they are > assigned in your context (there are different partitioning schemes) and > then you may have a better sense of it. > > I hope it helps > > Vinicius Scheidegger > > > > On Sat, Nov 14, 2020, 8:39 PM Mazen Ezzeddine < > mazen.ezzedd...@etu.univ-cotedazur.fr> wrote: > > > Given a business application that resorts into a message queue solution > > like Kafka, what is the best number of partitions to select for a given > > topic? what influences such a decision? > > > > > > On the other hand, say we want to achieve a maximal throughput of message > > consumption but at minimal resource consumption? what is the best number of > > topic consumers to configure statically? > > > > > > > > If dynamic scale up/down of topic consumers is enabled what would better : > > (1) start with one consumer and scale up consumers until a desired metric > > is achieved, or (2) start with consumers equal to the number of partitions > > and then scale down until the desired metric is achieved? > > > > > > Are you aware of any cloud provider that offers a message broker service > > (namely, Kafka) that support automatic scale of consumers? > > > > > > > > Thank you. > > > > >
