The second step can't be done through a LB. The produce will have to connect to the right broker directly. We don't do request forwarding in the brokers.
Also, I added https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowshouldIsetmetadata.broker.list? See if that answers your other question. Thanks, Jun On Mon, Mar 31, 2014 at 10:16 PM, kafka user <kafkaraic...@gmail.com> wrote: > Thanks Jun for your quick reply. > > If I understood correctly, for the first step ( fetching metadata ), > producer can still go through a VIP whereas for the second step ( once > producer gets the metadata ), producer needs to connect to the correct > instance of the broker that hosts the partition. I guess for the second > step, producer should be able to directly connect to the right broker and > this cant be achieved through a Load Balancer ( as LB may send the request > to any one of the machines in the pool ). > > Is my understanding correct ? Does it also mean that when a new broker is > added or an existing broker is removed, we need to keep the list of brokers > updated ? > > Thanks > > > > > > > > > > > On Mon, Mar 31, 2014 at 9:22 PM, Jun Rao <jun...@gmail.com> wrote: > > > In Kafka, the broker list you provide to the producer is used only for > > fetching metadata, which can be served on any broker. The producer then > > connects to the right brokers according to the metadata response. > > > > Thanks, > > > > Jun > > > > > > On Mon, Mar 31, 2014 at 6:04 PM, kafka user <kafkaraic...@gmail.com> > > wrote: > > > > > If we have a kafka producer connecting to a Kafka cluster behind a > > hardware > > > Load Balancer (VIP), will producer be able to send a message to a right > > > partition ? > > > Can one of the brokers in a cluster do a broker discovery to forward > the > > > message to ? > > > I guess my question is whether it makes sense to use a hardware load > > > balancer at all. > > > > > > Typical deployment model at our company is to have a pool of machines > and > > > LB decides which machine in a pool would receive a request. Looking to > > hear > > > from kafka experts what is the best way of setting up a cluster given > > that > > > the cluster has be behind a VIP. > > > > > > > > > Thanks > > > > > >
