Hello, I am working on developing a microservice based system which uses kafka as a messaging infrastructure. The microservices application are mainly kafka consumers and kafka streams applications and are deployed as docker containers on kubernetes.
The system should be designed to be auto scalable for which we are using Horizontal Pod Autoscaler feature of kubernetes which allows to instantiate more number of pods if a certain metric (e.g. cpu utilization) touches the threshold or reduce the pods in case the metric is way below the threshold. However, the problem is number of partitions in kafka are fixed so even if load on the system increases and the number of consumer pods are autoscaled, it could not be scaled beyond the number of partitions. So, after a point where number of pods is equal to number of partitions, the system can't be scaled beyond that. Is there a way to autoscale number of partitions also in kafka so the system can be auto scaled in cloud?
