Re: On the best number of partitions per topic for kafka based application and the best number of consumers per topic?

2020-11-15 Thread Mazen Ezzeddine
Perfect thanks, indeed I will be using HPA/Prometheus to achieve this dynamic consumer addition/removal for better resource consumption, while keeping up with the production rate of messages (perhaps using the consumer lag property). Best, From: Vinicius Scheidegg

Re: On the best number of partitions per topic for kafka based application and the best number of consumers per topic?

2020-11-15 Thread Vinicius Scheidegger
Kafka alone wouldn't scale down your consumers, but you can get metrics from Kafka and use them to change the number of consumers. Scaling up/down is normally done at other app levels, using tools whose purpose is to watch metrics and scaling up/down (say HPA kubernetes / openshift if you are work