Thanks, 
My question is mostly about dynamic resource optimization,

 say I configured my application with 30 partitions and then I managed for 30 
consumers (within a consumer group) to read/process the produced messages, but 
say at for instance at some unpeak load, I realized that a single consumer 
(within the same consumer group) could do consume all messages produced 
(production rate is low)? are you aware of any Kafka solution that would 
automatically join/leave consumers from within a consumer group dynamically 
e.g., based on how well the consumer group is doing to keep up with the 
production rate? 

On 2020/11/14 22:36:34, Vinicius Scheidegger <vinicius.scheideg...@gmail.com> 
wrote: 
> This depends on the design of your application and how you are using Kafka.
> 
> For instance, if you are using Kafka as a message queuing app, using
> consumers within a consumer group to load balance, then you should create
> the topics with as many partitions as the max number of consumers within
> the same consumer group reading from it.
> 
> For instance, in a regular load you believe you will have 8 consumers
> reading from a topic, but at peak you believe you could have around 30
> consumers. You should then have at least 30 partitions - each consumer in a
> consumer group is assigned to a partition. Having more consumers in a
> single consumer group than partitions are not effective as they won't be
> receiving any messages.
> 
> But as I mentioned earlier different designs may lead to different
> decisions. Try to understand first how partitions work. How they are
> assigned in your context (there are different partitioning schemes) and
> then you may have a better sense of it.
> 
> I hope it helps
> 
> Vinicius Scheidegger
> 
> 
> 
> On Sat, Nov 14, 2020, 8:39 PM Mazen Ezzeddine <
> mazen.ezzedd...@etu.univ-cotedazur.fr> wrote:
> 
> > Given a business application that resorts into a message queue solution
> > like Kafka, what is the best number of partitions to select for a given
> > topic? what influences such a decision?
> >
> >
> > On the other hand, say we want to achieve a maximal throughput of message
> > consumption but at minimal resource consumption? what is the best number of
> > topic consumers to configure statically?
> >
> >
> >
> > If dynamic scale up/down of topic consumers is enabled what would better :
> > (1) start with one consumer and scale up consumers until a desired metric
> > is achieved, or (2) start with consumers equal to the number of partitions
> > and then scale down until the desired metric is achieved?
> >
> >
> > Are you aware of any cloud provider that offers a message broker service
> > (namely, Kafka) that support automatic scale of consumers?
> >
> >
> >
> > Thank you.
> >
> >
> 

Reply via email to