Thanks for your review.
> Could you clarify the limitation of the current logic? The current logic cannot guarantee that the traffic of each bundle is the same, and must be balanced through split. However, the load of the topic is not always the same, and the traffic of the business changes with time, so the load of bundle will continue to change. If we rely on split+unload to balance, the number of bundles will eventually reach the upper limit. In order to avoid frequent split and unload, the current logic has many thresholds, allowing Broker to tolerate load imbalance, which is one of the reasons why the load gap between different nodes of the Pulsar cluster is large > For this issue, the community introduced a new assignment strategy, > LeastResourceUsageWithWeight, which better randomizes assignments. Yes, but LeastResourceUsageWithWeight still cannot completely solve the current problem, only alleviate it. We also optimized based on this implementation, but we will discuss this optimization in the following PIP, The current pip is not covered. > If each partition has the same load, then having the same number of topics per bundle should lead to the load balance. Then, I wonder how the current way, "hashing" does not achieve the goal here. We think that the loads of different partitions under a same topic are the same, but the loads of partitions of different topics are different. Bundles are shared by all topics in the entire namespace. If we guarantee each bundle has the same number of partitions, but these partitions may come from different topics, resulting in different loads for each bundle. If we split bundle according to load, the load of each topic may be different in different time periods, and it is impossible to keep the load of each Bundle the same. Using the round robin strategy, we can ensure that the number of partitions from a same Topic on each Bundle is roughly consistent, so that the load of each Bundle is also consistent. > happens if the leader restarts? how do we guarantee this mappingpersistence? 1）First of all, we need to find the starting bundle. partition-0 finds a bundle through consistent hashing, so as long as the number of bundles remains the same, the starting bundle is the same every time, and then other partitions 1, 2, 3, 4 ... is assigned the same result every time. 2）If the number of bundles changes, i.e. triggering split, the bundles of the entire namespace will be forced to be unloaded and all reassigned > It is unclear how RoundRobinPartitionAssigner will work with the existing > code. The specific implementation has been refined, please check the latest PIP issue On 2023/03/16 18:20:35 Heesung Sohn wrote: > Hi, > > Thank you for sharing this. > In general, I think this can be another good option for Pulsar load > assignment logic. > However, I have some comments below.
