> On Sept. 10, 2013, 9:46 p.m., Joel Koshy wrote: > > Following up on the performance concerns that Neha had raised - this will > > be a significant bottleneck for tools such as the mirror-maker as the > > rebalance latency will almost certainly multiply. > > > > We could consider utilizing the returned children information in > > ZkRebalancerListener's handleChildChange - i.e., curChilds will be the > > partitions if the parentPath is a topic. It won't be a trivial change then > > - we currently recompute the full map of topic/partitions during each > > rebalance. Now, we will need to "maintain" that data structure. There is > > also an initial bootstrap problem (that can also be addressed). > > > > This makes me wonder if it is better to implement selective rebalance - > > that way the overhead of going to ZK for partition information (for the > > affected topics alone) should be acceptable. It will also give us the added > > benefit of reducing the overall time for rebalance on a topic event. > >
Hi Joel, Thanks for the suggestion. I think this is a better fix for this bug, but this would require cache data that is read from ZK it would require longer stabilization and potential-bug-fix time. Since we are getting close to 0.8 final release I think it would be better to implement your proposal later in trunk. - Guozhang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14041/#review26022 ----------------------------------------------------------- On Sept. 10, 2013, 6:29 p.m., Guozhang Wang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/14041/ > ----------------------------------------------------------- > > (Updated Sept. 10, 2013, 6:29 p.m.) > > > Review request for kafka. > > > Bugs: KAFKA-1030 > https://issues.apache.org/jira/browse/KAFKA-1030 > > > Repository: kafka > > > Description > ------- > > Using the approach of reading directly from ZK. > > > Diffs > ----- > > core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala > e7a692a1d23ca5a9ecf86e3cb34be418b9c0c943 > > Diff: https://reviews.apache.org/r/14041/diff/ > > > Testing > ------- > > unit tests > > > Thanks, > > Guozhang Wang > >
