Re: Kafka partitioning and auto-scaling in k8s

2019-02-21 Thread Ali Nazemian
Hi Peter, Yes, I meant the data rate. The only issue is our application traffic is very fluctuated, so if I go with the high rate for the partition number it doesn't perform very well for the low data rate as it brings unnecessary network latency. I have found out sometimes the latency becomes hi

Re: Kafka partitioning and auto-scaling in k8s

2019-02-21 Thread Peter Bukowinski
I’ll assume when you say load, you mean data rate flowing into your kafka topic(s). One instance can consume from multiple partitions, so on a variable load workflow, it’s a good idea to have more partitions than your average workload will require. When the data rate is low, fewer consumers wil

Kafka partitioning and auto-scaling in k8s

2019-02-21 Thread Ali Nazemian
Hi All, I was wondering how an application can be auto-scalable if only a single instance can read from the single Kafka partition and two instances cannot read from the single partition at the same time with the same consumer group. Suppose there is an application that has 10 instances running o