Im wondering what the parameters are to instantiate a consumer rebalance. I have a topic that turns roughly 50K / minute across 6 partitions. Each is serviced by a separate dockerized consumer.
Roughly every 8-12 min this goes into a rebalance that may take up to a minute. When it returns it often puts some or all partitions on to a single consumer (leaving others idle). This may persist for a minute while it tries another arrangement. Eventually after 2-3 tries it will evenly distribute the partitions.. for a few minutes until it does another misguided attempt. As a result we have lag increasing from 0 to ~450K and back to 0 on a cycle. The data rate is assumed to be roughly consistent through these cycles. Resultant graph of lag is a sawtooth shape. Using 0.10.0.1 3 brokers Also - is there some way to set / control consumer 'assignment'? Or to 'suggest' a setting?
