Oh, Sriram, Thank you very much!
On Fri, Oct 11, 2013 at 5:44 PM, Sriram Subramanian < srsubraman...@linkedin.com> wrote: > We already have a JIRA for auto rebalance. I would be working on this soon. > > KAFKA-930 <https://issues.apache.org/jira/browse/KAFKA-930> > > > > On 10/11/13 5:39 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: > > >Hello Siyuan, > > > >For the automatic leader re-election, yes we are considering to make it > >work. Could you file a JIRA for this issue? > > > >For the high-level consumer's rebalancing logic, you can find it at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredictthere > >sultsoftheconsumerrebabalance%3F > > > >Guozhang > > > > > >On Fri, Oct 11, 2013 at 11:06 AM, hsy...@gmail.com <hsy...@gmail.com> > >wrote: > > > >> Hi Jun, > >> > >> Thanks for your reply, but in a real cluster, one broker could serve > >> different topics and different partitions, the simple consumer only has > >> knowledge of brokers that are available but it has no knowledge to > >>decide > >> which broker is best to pick up to consume messages. If you don't > >>choose > >> carefully, multiple simple consumer might end up with reading from same > >> node which is definitely not good for performance. > >> Interesting thing is I find out there is > >> command kafka-preferred-replica-election.sh which will try to equally > >> distribute the leadership among different brokers, this is good that I > >>can > >> always let my simple consumer reads from leader broker(even it fails, > >>the > >> replica will pick up as leader which is fine). But why don't kafka > >>cluster > >> run this command automatically when there is a broker change(up/down) in > >> the cluster so that the leadership can always be equally distributed > >>among > >> different brokers ASAP? I think it's very good for simple consumer to > >> decide which broker is good to read from. > >> > >> Another question is I'm also curious how high-level consumer is > >>balanced. I > >> assume each high-level consumer know other consumers(int the same group) > >> which broker they read message from and it can try to avoid those > >>brokers > >> and to pick up a free one? Is there a document for the balancing rule > >> among high-level consumer. Does it always guarantee that after several > >> leadership change/temporary broker fail, it can always equally > >>distribute > >> the read among the brokers. Basically I think it's nice to have a API to > >> let dev know which consumer reads from which broker otherwise I don't > >>know > >> anything behind the high-level consumer > >> > >> Thanks! > >> > >> Best, > >> Siyuan > >> > > > > > > > >-- > >-- Guozhang > >
