Hello Siyuan, Pre 0.8 Kafka does have a zookeeper based producer api, and this has been removed in 0.8. You can find some of the reasons in this JIRA:
https://issues.apache.org/jira/browse/KAFKA-369 Guozhang On Fri, Oct 11, 2013 at 5:16 PM, hsy...@gmail.com <hsy...@gmail.com> wrote: > That's why I'm asking, I would like to see a kafka zookeeper client api to > get TopicMetadata instead of my own hacky way to query the zookeeper > > Thanks! > Best, > Siyuan > > > On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues < > bruno.rodrig...@litux.org> wrote: > > > Why not ask zookeeper for the list of brokers and then ask a random > > broker for the metadata (and repeat if the broker is down), even if > > it's two calls. > > > > Heck it already does unnecessary connections. It connects to a broker, > > gets the metadata, disconnects, and then connects again for the data. > > If it's already assumed a producer or consumer will take some seconds > > until ready, what is another call gonna prejudice the flow. > > > > Then producers and consumers would then be consistently configured. Or > > allow the producers to also go to a broker instead of zookeeper. > > > > This way the consumer needs to know and hardcode at least one node. > > The node can fail. It can be changed. > > > > I thought zookeeper served to abstract this kind of complexity > > > > > > > > > > > > > > > > -- > > Bruno Rodrigues > > Sent from my iPhone > > > > No dia 11/10/2013, às 22:40, Neha Narkhede <neha.narkh...@gmail.com> > > escreveu: > > > > >>> For each consumer consumes different > > > topic/replica I have to specify those 20 brokers and go over all of > them > > to > > > know which broker is alive. And even worse how about I dynamically add > > new > > > broker into the cluster and remove the old one > > > > > > TopicMetadataRequest is a batch API and you can get metadata > information > > > for either a list of all topics or all topics in the cluster, if you > > > specify an empty list of topics. Adding a broker is not a problem since > > the > > > metadata request also returns the list of brokers in a cluster. The > > reason > > > this is better than reading from zookeeper is because the same > operation > > > would require multiple zookeeper roundtrips, instead of a single > > > TopicMetadataRequest roundtrip to some kafka broker. > > > > > > Thanks, > > > Neha > > > > > > > > >> On Fri, Oct 11, 2013 at 11:30 AM, hsy...@gmail.com <hsy...@gmail.com> > > wrote: > > >> > > >> Thanks guys! > > >> But I feel weird. Assume I have 20 brokers for 10 different topics > with > > 2 > > >> partitions and 2 replicas for each. For each consumer consumes > > different > > >> topic/replica I have to specify those 20 brokers and go over all of > > them to > > >> know which broker is alive. And even worse how about I dynamically add > > new > > >> broker into the cluster and remove the old one. I think it's nice to > > have a > > >> way to get metadata from zookeeper(centralized coordinator?) directly. > > >> > > >> Best, > > >> Siyuan > > >> > > >> > > >> On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede < > neha.narkh...@gmail.com > > >>> wrote: > > >> > > >>> If, for some reason, you don't have access to a virtual IP or load > > >>> balancer, you need to round robin once through all the brokers before > > >>> failing a TopicMetadataRequest. So unless all the brokers in your > > cluster > > >>> are down, this should not be a problem. > > >>> > > >>> Thanks, > > >>> Neha > > >>> > > >>> > > >>> On Thu, Oct 10, 2013 at 10:50 PM, hsy...@gmail.com <hsy...@gmail.com > > > > >>> wrote: > > >>> > > >>>> Hi guys, > > >>>> > > >>>> I'm trying to maintain a bunch of simple kafka consumer to consume > > >>> messages > > >>>> from brokers. I know there is a way to send TopicMetadataRequest to > > >>> broker > > >>>> and get the response from the broker. But you have to specify the > > >> broker > > >>>> list to query the information. But broker might not be available > > >> because > > >>> of > > >>>> some failure. My question is is there any api I can call and query > > >> broker > > >>>> metadata for topic/partition directly from zookeeper? I know I can > > >> query > > >>>> that information using zookeeper API. But that's not friendly > > >>> datastructure > > >>>> like the TopicMetadata/PartitionMetadata. Thank you! > > >>>> > > >>>> Best, > > >>>> Siyuan > > >> > > > -- -- Guozhang
