Hi everyone, We are running Kafka 0.8.1.1 with Storm. We wrote our own spout which uses the high level consumer API. Our setup is to create 4 spouts per worker. If your not familiar with Storm its basically 4 kafka consumers per java process. This particular consumer group is interested in 20 topics and ~150 partitions. When we increased the parallelism to 6 workers and 24 consumers we noticed certain consumers did not own any partitions in the group. These consumers were on certain hosts. We see their ephemeral nodes in zookeeper under their consumer group. We have also verified connectivity with kafka from those nodes. I also found if I can get those workers/consumers to run on the hosts with consumers that do own partitions they too will start owning partitions.
I'm also finding nothing in our logs in Kafka or on the consumer side which indicates any kind of problem. Any suggestions on what to try? Is it possible to force a consumer rebalance and who handles the partition assignment for a consumer group? Bryan
