What if the broker it tries to get the TopicMetadata from is down? I assume a broker is more like to be down than a zookeeper.
Thanks, Siyuan On Fri, Oct 11, 2013 at 2:39 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > >> For each consumer consumes different > topic/replica I have to specify those 20 brokers and go over all of them to > know which broker is alive. And even worse how about I dynamically add new > broker into the cluster and remove the old one > > TopicMetadataRequest is a batch API and you can get metadata information > for either a list of all topics or all topics in the cluster, if you > specify an empty list of topics. Adding a broker is not a problem since the > metadata request also returns the list of brokers in a cluster. The reason > this is better than reading from zookeeper is because the same operation > would require multiple zookeeper roundtrips, instead of a single > TopicMetadataRequest roundtrip to some kafka broker. > > Thanks, > Neha > > > On Fri, Oct 11, 2013 at 11:30 AM, hsy...@gmail.com <hsy...@gmail.com> > wrote: > > > Thanks guys! > > But I feel weird. Assume I have 20 brokers for 10 different topics with 2 > > partitions and 2 replicas for each. For each consumer consumes different > > topic/replica I have to specify those 20 brokers and go over all of them > to > > know which broker is alive. And even worse how about I dynamically add > new > > broker into the cluster and remove the old one. I think it's nice to > have a > > way to get metadata from zookeeper(centralized coordinator?) directly. > > > > Best, > > Siyuan > > > > > > On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <neha.narkh...@gmail.com > > >wrote: > > > > > If, for some reason, you don't have access to a virtual IP or load > > > balancer, you need to round robin once through all the brokers before > > > failing a TopicMetadataRequest. So unless all the brokers in your > cluster > > > are down, this should not be a problem. > > > > > > Thanks, > > > Neha > > > > > > > > > On Thu, Oct 10, 2013 at 10:50 PM, hsy...@gmail.com <hsy...@gmail.com> > > > wrote: > > > > > > > Hi guys, > > > > > > > > I'm trying to maintain a bunch of simple kafka consumer to consume > > > messages > > > > from brokers. I know there is a way to send TopicMetadataRequest to > > > broker > > > > and get the response from the broker. But you have to specify the > > broker > > > > list to query the information. But broker might not be available > > because > > > of > > > > some failure. My question is is there any api I can call and query > > broker > > > > metadata for topic/partition directly from zookeeper? I know I can > > query > > > > that information using zookeeper API. But that's not friendly > > > datastructure > > > > like the TopicMetadata/PartitionMetadata. Thank you! > > > > > > > > Best, > > > > Siyuan > > > > > > > > > >
