One of the big issues we run into with the 0.8 high-level consumer is the instability of partition assignments during a rebalance. The simple lexicographic assignment strategy means that if one consumer instance dies (and potentially a new instance with a different consumerId gets spun up to replace it) that every other consumer instance potentially changes its partition assignments. This is bad for data locality, cache-hit-rates, etc.
We'd really like it if under a rebalance consumers kept the same partition assignments as much as possible. So, if one instance goes away, another will have to take over that partition, but there's no reason that large numbers of consumers in the group should have to change what partition they consumer. Similarly, if a new instance is added, other consumers might relinquish excess partitions to maintain good balance of work-load across the group, but it would be nice if we didn't shuffle everything around. I've been working on an algorithm and implementation to support this, but before getting too far down the path, I wanted to check and see if anyone has previously developed a consumer that works like this. So far I haven't had any luck searching for one. Also, in reading the 0.9 consumer docs, there are some hints that rebalancing / partition assignment will work differently from the current assignment strategy (and, in particular, that it might be possible to plug in user-defined strategies). I'd love to know a little more about this, and if it might be simpler to achieve the behavior we're looking for with 0.9. Thanks, Kevin
