Oops. I forgot to mention another important item. I added it below(in bold).
Pros: - It supports more distributed load balance operations(bundle assignment) in a sequentially consistent manner - For really large clusters, by a partitioned system topic, BSC can be more scalable than the current single-leader coordination solution. - The load balance commands(across brokers) are sent via event sourcing(more reliable/transparent/easy-to-track) instead of RPC with retries. *- Bundle ownerships can be cached in the topic table-view from BSC. (no longer needs to store bundle ownership in metadata store(ZK))* Cons: - It is a new implementation and will require significant effort to stabilize the new implementation. (Based on our PoC code, I think the event sourcing handlers are easier to understand and follow the logic. Also, this new load balancer will be pluggable(will be implemented in new classes), so it should not break the existing load balance logic. Users will be able to configure old/new broker load balancer.) On Wed, Oct 19, 2022 at 10:17 AM Heesung Sohn <heesung.s...@streamnative.io> wrote: > Hi, > > On Wed, Oct 19, 2022 at 2:06 AM 丛搏 <congbobo...@gmail.com> wrote: > >> Hi, Heesung: >> I have some doubts. >> I review your PIP-192: New Pulsar Broker Load Balancer. I found that >> unload topic uses the leader broker to do, (Assigning, Return) uses >> the lookup request broker. why (Assigning, Return) not use a leader >> broker? >> I can think of a few reasons: >> 1. reduce leader broker pressure >> 2. does not strongly depend on the leader broker >> >> Yes, one of the goals of the PIP-192 is to distribute the load balance > logic to individual brokers (bundle assignment and bundle split). > > If (Assigning, Return) does not depend on the leader, it will bring > the following problems: > >> If (Assigning, Return) does not depend on the leader, it will bring >> the following problems: > > I assume what you meant by `(Assigning, Return) does not depend on the > leader` is the distributed topic assignment here(concurrent bundle > assignment across brokers). > > 1. leader clear bundle op and (Assigning, Return) will do at the same >> time, It will cause many requests to be retried, and the broker will >> be in chaos for a long time. > > I assume `leader clear bundle op` means bundle unloading, and `many > requests` means topic lookup requests(bundle assignment requests). > The leader unloads only high-loaded bundles in the "Owned" state. So, the > leader does not unload bundles that are in the process of assignment states. > Even if there are conflict state changes, only the first valid state > change will be accepted(as explained in Conflict State Resolution(Race > Conditions section in the PIP)) in BSC. > > Also, another goal of this PIP-192 is to reduce client lookup retries. In > BSC, the client lookup response will be deferred(max x secs) until the > bundle state becomes finally "Owned". > > > > >> 2. bundle State Channel(BSC) owner depends on the leader broker, this >> also makes topic transfer strongly dependent on the leader. >> > BSC will use separate leader znodes to decide the owner brokers of the > internal BSC system topic.As described in this section in the PIP-192, > "Bundle State and Load Data TableView Scalability", > We could use a partitioned topic(configurable) for this BSC system topic. > Then, there could be a separate owner broker for each partition > (e.g. zk leader znodes, /loadbalance/leader/part-1-owner, part-2-owner, > ..etc). > > > >> 3. the code becomes more complex and harder to maintain >> >> What tradeoffs are the current implementations based on? >> >> Here are some Pros and Cons of BSC I can think of. > > Pros: > - It supports more distributed load balance operations(bundle assignment) > in a sequentially consistent manner > - For really large clusters, by a partitioned system topic, BSC can be > more scalable than the current single-leader coordination solution. > - The load balance commands(across brokers) are sent via event > sourcing(more reliable/transparent/easy-to-track) instead of RPC with > retries. > > Cons: > - It is a new implementation and will require significant effort to > stabilize the new implementation. > (Based on our PoC code, I think the event sourcing handlers are easier to > understand and follow the logic. > Also, this new load balancer will be pluggable(will be implemented in new > classes), so it should not break the existing load balance logic. > Users will be able to configure old/new broker load balancer.) > > > Thank you for sharing your questions about PIP-192 here. But I think this > PIP-215 is independent of PIP-192(though PIP-192 needs some of the features > in PIP-215). > > Thanks, > Heesung > > > > > >> Thanks, >> bo >> >> Heesung Sohn <heesung.s...@streamnative.io.invalid> 于2022年10月19日周三 >> 07:54写道： >> > >> > Hi pulsar-dev community, >> > >> > I raised a pip to discuss : PIP-215: Configurable Topic Compaction >> Strategy >> > >> > PIP link: https://github.com/apache/pulsar/issues/18099 >> > >> > Regards, >> > Heesung >> >
