Hello Siyuan, For the automatic leader re-election, yes we are considering to make it work. Could you file a JIRA for this issue?
For the high-level consumer's rebalancing logic, you can find it at https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredicttheresultsoftheconsumerrebabalance%3F Guozhang On Fri, Oct 11, 2013 at 11:06 AM, hsy...@gmail.com <hsy...@gmail.com> wrote: > Hi Jun, > > Thanks for your reply, but in a real cluster, one broker could serve > different topics and different partitions, the simple consumer only has > knowledge of brokers that are available but it has no knowledge to decide > which broker is best to pick up to consume messages. If you don't choose > carefully, multiple simple consumer might end up with reading from same > node which is definitely not good for performance. > Interesting thing is I find out there is > command kafka-preferred-replica-election.sh which will try to equally > distribute the leadership among different brokers, this is good that I can > always let my simple consumer reads from leader broker(even it fails, the > replica will pick up as leader which is fine). But why don't kafka cluster > run this command automatically when there is a broker change(up/down) in > the cluster so that the leadership can always be equally distributed among > different brokers ASAP? I think it's very good for simple consumer to > decide which broker is good to read from. > > Another question is I'm also curious how high-level consumer is balanced. I > assume each high-level consumer know other consumers(int the same group) > which broker they read message from and it can try to avoid those brokers > and to pick up a free one? Is there a document for the balancing rule > among high-level consumer. Does it always guarantee that after several > leadership change/temporary broker fail, it can always equally distribute > the read among the brokers. Basically I think it's nice to have a API to > let dev know which consumer reads from which broker otherwise I don't know > anything behind the high-level consumer > > Thanks! > > Best, > Siyuan > -- -- Guozhang
