Hi Jun, Thanks for your reply, but in a real cluster, one broker could serve different topics and different partitions, the simple consumer only has knowledge of brokers that are available but it has no knowledge to decide which broker is best to pick up to consume messages. If you don't choose carefully, multiple simple consumer might end up with reading from same node which is definitely not good for performance. Interesting thing is I find out there is command kafka-preferred-replica-election.sh which will try to equally distribute the leadership among different brokers, this is good that I can always let my simple consumer reads from leader broker(even it fails, the replica will pick up as leader which is fine). But why don't kafka cluster run this command automatically when there is a broker change(up/down) in the cluster so that the leadership can always be equally distributed among different brokers ASAP? I think it's very good for simple consumer to decide which broker is good to read from.
Another question is I'm also curious how high-level consumer is balanced. I assume each high-level consumer know other consumers(int the same group) which broker they read message from and it can try to avoid those brokers and to pick up a free one? Is there a document for the balancing rule among high-level consumer. Does it always guarantee that after several leadership change/temporary broker fail, it can always equally distribute the read among the brokers. Basically I think it's nice to have a API to let dev know which consumer reads from which broker otherwise I don't know anything behind the high-level consumer Thanks! Best, Siyuan
