See below On Tue, Feb 23, 2016 at 11:45 AM, vivek shankar <vivekshanka...@gmail.com> wrote:
> Hello All, > > Can you please help with the below : > > I was reading up on Kafka 0.9 API version and came across the below : > > The following is a draft design that uses a high-available consumer > coordinator at the broker side to handle consumer rebalance. By migrating > the rebalance logic from the consumer to the coordinator we can resolve the > consumer split brain problem and help thinner the consumer client. > > Please can you help with the below questions : > > I have a single topic with single consumer group id. There are 20 > partitions. The consumer client is running on a single node which has 20 > threads. We are using zookeeper to fetch messages from Kafka. > > 1) When using Kafka 0.8.2 API , how can re balancing be done? Can the same > consumer project be deployed into multiple nodes and will zookeeper > automatically do the rebalancing? > 0.8.2 has two consumer APIs. The high level consumer (Also known as ZookeeperConsumerConnector) has the concept of Consumer Groups. If the "same consumer" is deployed on different nodes and they all have the same group.id, the consumers will discover each other via zookeeper and balance consumption of partitions between them. > > 2) What are the changes that we need to take care of when upgrading to 0.9 > API version? Should underlying Kafka installation be changed by itself or > is it suffice to just use 0.9 consumer api? > Apache Kafka upgrades are always broker first, so you much upgrade the Kafka brokers before using the 0.9 API on the clients. The 0.9 API is completely different, so you will also need to rewrite quite a bit of code when upgrading. > > 3) Can zookeeper be completely eliminated when upgrading to 0.9 version? > In 0.9. clients no longer need to connect to Zookeeper. The brokers still use Zookeeper for things like topic configuration, controller election, etc. > > Thank You! > > Regards > Hope this helps, Gwen
