In 0.8, we got rid of zookeeper from the producer and replaced it with a metadata API. What is the API like? Can we call the API manually? On startup and whenever a request fails, the producer refreshes its view of the cluster by sending a metadata request to any of the brokers. But in my test, in Kafka 0.8, if the broker.list for a producer consist of broker1 and broker2, and broker 1 and broker 2 is not startup ,only broker 3 start, then if the producer start, it can’t send any data for a new topic correctly.It says, fetching topic metadata for topics from broker 1 and broker 2 failed. On the other hand If the broker.list consist of broker 1 and broker 2, and broker 1 and broker 3 are working. When the producer start to send some data for a new topic, there will be two partitions in broker 1 and two in broker 3.(The num.partitions configured in all brokers are 4). So we must guarantee at least one broker in the broker.list are working normally?Right? Why you get rid of zookeeper from the producer? With Zookeeper(like kafka 0.7.2), the producer have no this restriction.
From: Neha Narkhede [mailto:neha.narkh...@gmail.com] Sent: 2013年1月15日 13:59 To: dev@kafka.apache.org Cc: Jun Guo -X (jungu - CIIC at Cisco) Subject: Re: About kafka 0.8 producer zookeeper-based load balancing on per-request basis Than how it achieve zookeeper-based load balance on per-request basis? You are asking two related but different questions. One is how does load balancing work and the other is how does broker discovery works. Jun explained how load balancing works and how requests are routed to partitons. In 0.7, there were 2 options for broker discovery - zookeeper and hardware load balancer (VIP). The zookeeper based producer got notified by zookeeper whenever a new broker came up or an existing broker went down. In 0.8, we got rid of zookeeper from the producer and replaced it with a metadata API. On startup and whenever a request fails, the producer refreshes its view of the cluster by sending a metadata request to any of the brokers. So if a broker goes down or a new broker comes up, the leader for some partitions might change and the producer will know since its requests to the older leaders will fail. Hope that helps, Neha On Mon, Jan 14, 2013 at 9:52 PM, Jun Rao <jun...@gmail.com<mailto:jun...@gmail.com>> wrote: Basically, we spread partitions among multiple brokers. If a message is sent without a key, the producer picks a random partition to balance the load. If a message has a key, the default partitioner hashes the key to one of the partitions deterministically. Then, the load may not always be balanced. Thanks, Jun On Mon, Jan 14, 2013 at 9:35 PM, Jun Guo -X (jungu - CIIC at Cisco) < ju...@cisco.com<mailto:ju...@cisco.com>> wrote: > Hi, > We know, in kafka 0.8, producer connect to broker directly, it without > connecting to zookeeper. Than how it achieve zookeeper-based load balance > on per-request basis? > Actually, when a topic be created, its partition will distributed in one > or more brokers. When a message be sent, it will be delivered to a certain > partition according to its key word. That is to say ,a certain must be sent > to a fixed partition on a fixed broker. How the so called load balancing > works? > > Best Regards >
