[
https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13769879#comment-13769879
]
Guozhang Wang commented on KAFKA-1030:
--------------------------------------
Here are the performance testing results:
Setup: 1) 5 instances of mirror maker consuming from around 3800
topic/partitions, 2) 1 instance of console consumer consuming from around 300
topic/partitions.
1). Bouncing mirror makers:
ZK-located-in-same-DC: 4 minutes and 20 seconds with the fix
ZK-located-in-same-DC: 3 minutes 50 secs without the fix
ZK-located-in-other-DC: 8 minutes 2 seconds with the fix
ZK-located-in-other-DC: 7 minutes 6 seconds without the fix
2). Bouncing console consumer
ZK-located-in-same-DC: 15 seconds with the fix
ZK-located-in-same-DC: 15 seconds without the fix
---------------
Given the results, I think it worth pushing this approach (read-from-ZK) in 0.8
and we can later pursue the other approach Joel proposed in the reviewboard in
trunk.
> Addition of partitions requires bouncing all the consumers of that topic
> ------------------------------------------------------------------------
>
> Key: KAFKA-1030
> URL: https://issues.apache.org/jira/browse/KAFKA-1030
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Reporter: Swapnil Ghike
> Assignee: Guozhang Wang
> Priority: Blocker
> Fix For: 0.8
>
> Attachments: KAFKA-1030-v1.patch
>
>
> Consumer may not notice new partitions because the propagation of the
> metadata to servers can be delayed.
> Options:
> 1. As Jun suggested on KAFKA-956, the easiest fix would be to read the new
> partition data from zookeeper instead of a kafka server.
> 2. Run a fetch metadata loop in consumer, and set auto.offset.reset to
> smallest once the consumer has started.
> 1 sounds easier to do. If 1 causes long delays in reading all partitions at
> the start of every rebalance, 2 may be worth considering.
>
> The same issue affects MirrorMaker when new topics are created, MirrorMaker
> may not notice all partitions of the new topics until the next rebalance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
