shirenchuang commented on a change in pull request #11453:
URL: https://github.com/apache/kafka/pull/11453#discussion_r741666574
##########
File path: core/src/main/scala/kafka/server/ZkAdminManager.scala
##########
@@ -152,7 +152,7 @@ class ZkAdminManager(val config: KafkaConfig,
responseCallback: Map[String, ApiError] => Unit): Unit = {
// 1. map over topics creating assignment and calling zookeeper
- val brokers = metadataCache.getAliveBrokers()
+ val brokers = metadataCache.getAliveBrokers().toList.sortBy(_.id)
Review comment:
您好! 非常抱歉打扰到您了, 昨天我回复您说我可以把ReplicationControlManager那里的排序问题一起修复掉,
但是我今天看了一下,我对kraft这一块的代码还不是很熟,所以您能创建一个新的jira来跟踪一下吗？
还有我简单的描述一下这个问题为什么需要被修复
虽然这是一个比较不容易发现和影响比较小的bug
Hello! I am very sorry to disturb you. Yesterday I replied to you that I can
fix the sorting problem in ReplicationControlManager together, but I took a
look today. I am not very familiar with the code of kraft, so can you create A
new jira to follow up?
And let me briefly describe why this problem needs to be fixed
Although this is a bug that is not easy to find and has a relatively small
impact
之前我有针对这个bug写过一篇文章,写的比较详细
[关于分区副本分配相关的Bug...](https://www.szzdzhp.com/kafka/other/maybe-bug1.html)
但是是中文的，所以我在这里用英文简单描述一下这个问题
I have written an article about this bug before, and I wrote it in more
detail.
[Bugs related to partition copy
allocation...](https://www.szzdzhp.com/kafka/other/maybe-bug1.html)
But it’s in Chinese, so I’ll briefly describe this problem in English here.
通过分析得知, 需要进行分区副本分配的操作有下面三种
1. 创建Topic
2. 扩分区
3. 分区副本重分配
上面除了 1.创建Topic的时候 没有进行排序, 2,3 都进行了排序
Through analysis, we know that there are the following three operations that
require partition copy allocation
1. Create Topic
2. Expand Partition
3. Partition copy redistribution
Except for the above 1. When creating the Topic, no sorting is performed, 2
and 3 are all sorted
通过跟王国璋老师的沟通,貌似官方期望的是 创建Topic的也是需要进行排序的, 所以基本上确定了这么一个bug.
Through communication with Teacher Wang Guozhang, it seems that the official
expectation is that the creation of Topic also needs to be sorted, so such a
bug is basically determined.
接下来我来重现一下bug,以及解决验证这个bug
Next, let me reproduce the bug and solve the verification bug
## Check the Bug
**I added a little log to the partition copy distribution rules to observe
the distribution**
**第一步**.创建一个3分区 单副本的 Topic
**setp 1**. create a topic
`--create --bootstrap-server xxxxx --partitions 3 --replication-factor 1
--topic szz2`
**the brokerArray is not sorted.**
**第二步. 对这个topic扩分区到5个分区**
**setp2.** `execute kafka.admin.TopicCommand --alter --partitions 5
--topic szz2`
最后我们看看分配结果
Finally, we look at the distribution results
Broker-0 被分到了两个, Broker-3却一个也没有分配到, 假如一开始创建topic的时候就已经是排序了,就不会出现这种情况
Broker-0 is divided into two, but Broker-3 is not assigned to one. If the
topic is already sorted when the topic is created at the beginning, this will
not happen
---
并且当我们进行 分区副本重分配的时候,可以看到 这个也是排序的, 只有创建Topic没有排序
**And when we redistribute partition copies, we can see that this is also
sorted, only the creation of Topic is not sorted**
## Where is the bug
这里拿到的就是无序的,处理updateMetadata请求的时候 拿到的请求参数就是无序的,所以这里也是无序的
当然我们也可以修复一下这里，在处理updateMetadata的时候将它变成有序的，不过我不确定全局的liveBroker变成有序之后会不会影响其他地方，所以解决这个问题，我觉得还是在创建Topic的时候进行排序，既可以解决bug，又可以保证不会引入其他可能存在的bug
What you get here is disordered. When processing updateMetadata requests,
the request parameters you get are disordered, so here is also disordered.
Of course, we can also fix this and make it orderly when processing
updateMetadata, but I am not sure whether the global liveBroker will affect
other places after it becomes orderly, so to solve this problem, I think it is
still creating Sorting topics in topics can not only solve bugs, but also
ensure that other possible bugs will not be introduced
## 解决Bug
我们只需要在创建Topic的时候 排序就行了
We only need to sort when creating Topic
看看解决后的结果
1. 创建Topic的时候已经排序了
2. 最终分配结果是 {0,1,2,3,4}每个Broker都分配到了一个分区, 分区就达到均衡的目的了
See the result after the solution
1. Sorted when creating Topic
2. The final allocation result is {0,1,2,3,4} Each Broker is allocated to a
partition, and the partition achieves the goal of balance
因为改动非常小, 我就不单独写一个 unit test 了。
Because the changes are very small, I won't write a separate unit test.
期待收到您的回复
Looking forward to receiving your reply
##########
File path: core/src/main/scala/kafka/server/ZkAdminManager.scala
##########
@@ -152,7 +152,7 @@ class ZkAdminManager(val config: KafkaConfig,
responseCallback: Map[String, ApiError] => Unit): Unit = {
// 1. map over topics creating assignment and calling zookeeper
- val brokers = metadataCache.getAliveBrokers()
+ val brokers = metadataCache.getAliveBrokers().toList.sortBy(_.id)
Review comment:
您好! 非常抱歉打扰到您了, 昨天我回复您说我可以把ReplicationControlManager那里的排序问题一起修复掉,
但是我今天看了一下,我对kraft这一块的代码还不是很熟,所以您能创建一个新的jira来跟踪一下吗？
还有我简单的描述一下这个问题为什么需要被修复
虽然这是一个比较不容易发现和影响比较小的bug
Hello! I am very sorry to disturb you. Yesterday I replied to you that I can
fix the sorting problem in ReplicationControlManager together, but I took a
look today. I am not very familiar with the code of kraft, so can you create A
new jira to follow up?
And let me briefly describe why this problem needs to be fixed
Although this is a bug that is not easy to find and has a relatively small
impact
之前我有针对这个bug写过一篇文章,写的比较详细
[关于分区副本分配相关的Bug...](https://www.szzdzhp.com/kafka/other/maybe-bug1.html)
但是是中文的，所以我在这里用英文简单描述一下这个问题
I have written an article about this bug before, and I wrote it in more
detail.
[Bugs related to partition copy
allocation...](https://www.szzdzhp.com/kafka/other/maybe-bug1.html)
But it’s in Chinese, so I’ll briefly describe this problem in English here.
通过分析得知, 需要进行分区副本分配的操作有下面三种
1. 创建Topic
2. 扩分区
3. 分区副本重分配
上面除了 1.创建Topic的时候 没有进行排序, 2,3 都进行了排序
Through analysis, we know that there are the following three operations that
require partition copy allocation
1. Create Topic
2. Expand Partition
3. Partition copy redistribution
Except for the above 1. When creating the Topic, no sorting is performed, 2
and 3 are all sorted
通过跟王国璋老师的沟通,貌似官方期望的是 创建Topic的也是需要进行排序的, 所以基本上确定了这么一个bug.
Through communication with Teacher Wang Guozhang, it seems that the official
expectation is that the creation of Topic also needs to be sorted, so such a
bug is basically determined.
接下来我来重现一下bug,以及解决验证这个bug
Next, let me reproduce the bug and solve the verification bug
## Check the Bug
**I added a little log to the partition copy distribution rules to observe
the distribution**
**第一步**.创建一个3分区 单副本的 Topic
**setp 1**. create a topic
`--create --bootstrap-server xxxxx --partitions 3 --replication-factor 1
--topic szz2`
**the brokerArray is not sorted.**
**第二步. 对这个topic扩分区到5个分区**
**setp2.** `execute kafka.admin.TopicCommand --alter --partitions 5
--topic szz2`
最后我们看看分配结果
Finally, we look at the distribution results
Broker-0 被分到了两个, Broker-3却一个也没有分配到, 假如一开始创建topic的时候就已经是排序了,就不会出现这种情况
Broker-0 is divided into two, but Broker-3 is not assigned to one. If the
topic is already sorted when the topic is created at the beginning, this will
not happen
---
并且当我们进行 分区副本重分配的时候,可以看到 这个也是排序的, 只有创建Topic没有排序
**And when we redistribute partition copies, we can see that this is also
sorted, only the creation of Topic is not sorted**
## Where is the bug
这里拿到的就是无序的,处理updateMetadata请求的时候 拿到的请求参数就是无序的,所以这里也是无序的
当然我们也可以修复一下这里，在处理updateMetadata的时候将它变成有序的，不过我不确定全局的liveBroker变成有序之后会不会影响其他地方，所以解决这个问题，我觉得还是在创建Topic的时候进行排序，既可以解决bug，又可以保证不会引入其他可能存在的bug
What you get here is disordered. When processing updateMetadata requests,
the request parameters you get are disordered, so here is also disordered.
Of course, we can also fix this and make it orderly when processing
updateMetadata, but I am not sure whether the global liveBroker will affect
other places after it becomes orderly, so to solve this problem, I think it is
still creating Sorting topics in topics can not only solve bugs, but also
ensure that other possible bugs will not be introduced
## 解决Bug
我们只需要在创建Topic的时候 排序就行了
We only need to sort when creating Topic
看看解决后的结果
1. 创建Topic的时候已经排序了
2. 最终分配结果是 {0,1,2,3,4}每个Broker都分配到了一个分区, 分区就达到均衡的目的了
See the result after the solution
1. Sorted when creating Topic
2. The final allocation result is {0,1,2,3,4} Each Broker is allocated to a
partition, and the partition achieves the goal of balance
因为改动非常小, 我就不单独写一个 unit test 了。
Because the changes are very small, I won't write a separate unit test.
期待收到您的回复
Looking forward to receiving your reply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
