+1 (binding) Regarding the questions from Heesung, the customized load manager could implement its own lookup logic when the `LookupOptions` has a specific property.
For example, given a topic "my-topic" and two brokers with "lookup.broker" as "A" and "B". - The client with lookup property "broker=A" will receive the response that the owner is broker-0 - The client with lookup property "broker=B" will receive the response that the owner is broker-1 - The client without any lookup property will receive the response that the owner is the bundle's owner determined by the built-in load manager Thanks, Yunze On Tue, Jul 30, 2024 at 7:22 PM Zike Yang <z...@apache.org> wrote: > > Hi everyone, Thanks for your comments. > > > 1. Given that the current pulsar assigns topics to brokers at the > bundle level, what happens with a topic lookup with rack info(rack A) > for the bundle that's already assigned to a different rack(rack B)? > > In this case, the bundle has already been assigned to the specific > broker. Then, the rack info for other clients shouldn't be respected. > The customized load manager(rack-aware load manager) will still > redirect the topic lookup to the rack B broker. > However, an improved implementation would be to decide the best rack > to act as the owner based on the current load conditions. > > > 2. Given that the latest load shedders(TransferShedder and AvgShedder) > determine the destination brokers from the leader, how would the > leader make sure to unload bundles(topics) to the brokers in the same > rack? > > The rack info could be treated as an affinity property. When facing > load imbalance, rack-info-based decisions may need to yield to > load-based decisions. > > The rack-aware lookup scenario is an example of usage for this > proposal. This proposal will not cover the detailed implementation of > the rack-aware load manager. It focuses only on exposing certain > interface capabilities to enable features like rack-aware lookup > functionality. > > Thanks, > Zike Yang > > On Sat, Jul 27, 2024 at 12:20 AM Heesung Sohn <hees...@apache.org> wrote: > > > > Although I agree with this direction, having the config in the lookup > > requests, I have the following questions. > > > > 1. Given that the current pulsar assigns topics to brokers at the > > bundle level, what happens with a topic lookup with rack info(rack A) > > for the bundle that's already assigned to a different rack(rack B)? > > 2. Given that the latest load shedders(TransferShedder and AvgShedder) > > determine the destination brokers from the leader, how would the > > leader make sure to unload bundles(topics) to the brokers in the same > > rack? > > > > > > Thanks, > > Heesung > > > > On Fri, Jul 26, 2024 at 8:29 AM Heesung Sohn <hees...@apache.org> wrote: > > > > > > +1 (not binding) > > > > > > Heesung > > > > > > On Thu, Jul 25, 2024 at 7:52 PM Jie crossover <crossover...@gmail.com> > > > wrote: > > > > > > > > +1 > > > > I left some comments in the PR. > > > > -- > > > > Best Regards！ > > > > crossoverJie > > > > > > > > > > > > Zike Yang <z...@apache.org> 于2024年7月25日周四 11:15写道： > > > > > > > > > Hi, all, > > > > > > > > > > I proposed a new proposal to add client properties support for the > > > > > lookup: https://github.com/apache/pulsar/pull/23075 > > > > > > > > > > PTAL and share your thoughts. Thanks! > > > > > > > > > > BR, > > > > > Zike Yang > > > > >
