[
https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621456#comment-13621456
]
Neha Narkhede edited comment on KAFKA-330 at 4/4/13 12:08 AM:
--------------------------------------------------------------
Let's do some zookeeper math here to see how long it takes to delete one topic,
8 partitions from a 6 node kafka cluster -
# of zk ops operation during delete topic
1 val partitionAssignment =
ZkUtils.getPartitionAssignmentForTopics(zkClient, topics.toSeq)
7 val brokers =
ZkUtils.getAllBrokersInCluster(zkClient)
1 ZkUtils.getAllReplicasOnBroker(zkClient,
topics.toSeq, brokers.map(_.id)) (This is a redundant read from
zookeeper, so reuse the info read in step 1)
2 removeReplicaFromIsr ->
getLeaderIsrAndEpochForPartition, conditionalUpdatePersistentPath
9 removeFromTopicsBeingDeleted ->
readDataMaybeNull (1), deletePath (8)
20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that
kafka consumers and brokers share does in best case), that is 200ms per topic
With 50 such topics, it is 10 seconds. That is the amount of time you are
starving other partitions from being available!
What you can do, for simplicity purposes, is keep the existing long lock on the
controller side for this patch. We can improve it later or in 0.8.1
Also, the log side of your patch does not acquire the lock. You used the delete
APIs that were used by unit tests so far. So they don't deal with the issues
I've mentioned above in my comments.
Regarding LogManager - Let's look at the modified version of your patch and see
if that solves the problems I've outlined above wrt to interleaving other
operations with delete log.
was (Author: nehanarkhede):
Let's do some zookeeper math here to see how long it takes to delete one
topic, 8 partitions from a 6 node kafka cluster -
# of zk ops operation during delete topic
1 val partitionAssignment =
ZkUtils.getPartitionAssignmentForTopics(zkClient, topics.toSeq)
7 val brokers =
ZkUtils.getAllBrokersInCluster(zkClient)
1 ZkUtils.getAllReplicasOnBroker(zkClient,
topics.toSeq, brokers.map(_.id)) (This is a redundant read from
zookeeper, so reuse the info read in step 1)
2 removeReplicaFromIsr ->
getLeaderIsrAndEpochForPartition, conditionalUpdatePersistentPath
9 removeFromTopicsBeingDeleted ->
readDataMaybeNull (1), deletePath (8)
20 zookeeper ops. With 10ms per op, (which is a what a zookeeper cluster that
kafka consumers and brokers share does in best case), that is 200ms per topic
With 50 such topics, it is 10 seconds. That is the amount of time you are
starving other partitions from being available!
Also, the log side of your patch does not acquire the lock. You used the delete
APIs that were used by unit tests so far. So they don't deal with the issues
I've mentioned above in my comments.
Regarding LogManager - Let's look at the modified version of your patch and see
if that solves the problems I've outlined above wrt to interleaving other
operations with delete log.
> Add delete topic support
> -------------------------
>
> Key: KAFKA-330
> URL: https://issues.apache.org/jira/browse/KAFKA-330
> Project: Kafka
> Issue Type: Bug
> Components: controller, log, replication
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Swapnil Ghike
> Priority: Blocker
> Labels: features, kafka-0.8, p2, project
> Fix For: 0.8
>
> Attachments: kafka-330-v1.patch
>
>
> One proposal of this API is here -
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+replication+detailed+design+V2#KafkareplicationdetaileddesignV2-Deletetopic
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
