Both the producer and consumer clients will take care of rebalancing if there is a broker failure. The consumer handles this entirely in Zookeeper.
For the producer side, you do need some sort of front end to handle the metadata requests, or else you would constantly have to change your producer configuration if brokers fail. In our case, we use a load balancer for this because it's the most robust solution. Only the metadata request itself will go through the load balancer. Once the producer has the metadata for the topic and partition, it reconnects directly to the leader for the partition for producing. -Todd On 3/11/14 10:07 PM, "Cassa L" <lcas...@gmail.com> wrote: >Hi, > Is there documentation on how connection management, connection pooling >works in Kafka for multiple brokers? Do I need to take care of my own >connection management for cluster? e.g. while publishing messages, if one >broker stops responding, I need to switch to the other. Does Kafka client >takes care of this or do I need load balancer between brokers and client? > >Thanks, >LCassa
