Thanks. I checked in my sample code that requests have been distributed to brokers evenly without load balancers.
..Leena On Wed, Mar 12, 2014 at 3:34 PM, Todd Palino <tpal...@linkedin.com> wrote: > I apologize, you are correct. I forgot that you can pass in a list of > hosts. Using a balancer is the way we handle it here, mostly because we > have many producers, and they are not all under our direct control. > > To answer the question of what load balancer, we have hardware load > balancers in place in our environment. I'm not certain of the model, as I > generally don't interface with them directly (that's the purview of the > network operations team). > > -Todd > > On 3/12/14 1:47 PM, "Cassa L" <lcas...@gmail.com> wrote: > > >Hi Todd, > > I am not very clear about you mentioned - "Both the producer and > >consumer > >clients will take care of rebalancing if > >there is a broker failure." " > >For the producer side, you do need some sort of front end to handle the > >metadata requests, or else you would constantly have to change your > >producer configuration if brokers fail." > > > > Isn't broker client API needs to take care of collecting metadata based > >on > >the list of brokers that I pass in producer API? Why would I need load > >balancer just to get metadata? > > > >Regards, > >LCass > > > > > >On Wed, Mar 12, 2014 at 2:39 AM, Atig <andthereitg...@hotmail.com> wrote: > > > >> Hello > >> > >> Can you elaborate on what is your producer setup in terms of fetching > >>the > >> metadata (in conjunction with a load balancer) and some sort of code > >> skeleton? New to Kafka here. > >> > >> Also, if you don't mind sharing what do you use as load balancer in this > >> scenario? > >> > >> Thanks > >> > >> --- Original Message --- > >> > >> From: "Todd Palino" <tpal...@linkedin.com> > >> Sent: 12 March 2014 07:11 > >> To: users@kafka.apache.org > >> Subject: Re: Kafka Cluster management > >> > >> Both the producer and consumer clients will take care of rebalancing if > >> there is a broker failure. The consumer handles this entirely in > >>Zookeeper. > >> > >> For the producer side, you do need some sort of front end to handle the > >> metadata requests, or else you would constantly have to change your > >> producer configuration if brokers fail. In our case, we use a load > >> balancer for this because it's the most robust solution. Only the > >>metadata > >> request itself will go through the load balancer. Once the producer has > >> the metadata for the topic and partition, it reconnects directly to the > >> leader for the partition for producing. > >> > >> -Todd > >> > >> On 3/11/14 10:07 PM, "Cassa L" <lcas...@gmail.com> wrote: > >> > >> >Hi, > >> > Is there documentation on how connection management, connection > >>pooling > >> >works in Kafka for multiple brokers? Do I need to take care of my own > >> >connection management for cluster? e.g. while publishing messages, if > >>one > >> >broker stops responding, I need to switch to the other. Does Kafka > >>client > >> >takes care of this or do I need load balancer between brokers and > >>client? > >> > > >> >Thanks, > >> >LCassa > >> > >> > >
