Hi Kai, Good feature. Please add more info about how to enable this in an existing cluster. Do we plan to write bundle data to both zk and system topic during rolling update?
Thanks, Haiting On 2022/04/07 01:35:09 Kai Wang wrote: > Hi Pulsar community, > > I created a PIP to use the system topic to store the bundle load data in the > load manager. > > The proposal can be found: https://github.com/apache/pulsar/issues/15037 > > --------- > ## Motivation > > Currently, Pulsar LoadManager is using Zookeeper to store the bundle load > data, when we have many bundles, this might put a lot of pressure on > Zookeeper. > > ## Goal > > This PIP proposes storing the load manager's bundle load data to a system > topic and using TableView to read it. > > > > ## Implementation > > Since the bundle load data is stats data, it no needs a strong consistent > guarantee. So we can use the system topic to store the load data and use > TableView to read it. > > > ### System topic client > > > > Add a new SystemTopicClient calls `LoadBalanceBundleDataSystemTopicClient` , > the topic name is `persistent://pulsar/system/__load_balance_bundle_data`, > and the key is the bundle name, value is BundleData. > > ```java > public class LoadBalanceBundleDataSystemTopicClient extends > SystemTopicClientBase<LoadBalanceBundleDataEvent> { > // ... > } > ``` > > > > Add new Event calls `LoadBalanceBundleDataEvent` > > ``` > @Data > public class LoadBalanceBundleDataEvent { > > private String bundle; > > private BundleData bundleData; > } > > ``` > > > ### ModularLoadManagerImpl > > > Add a TableView in `ModularLoadManagerImpl` to replace the bundle cache > > ``` > private TableView<String, BundleData> bundlesTableView; > ``` > > > ## Compatibility > > This feature can have both backward and forward compatibility since the > bundle data is stats data. > > --------- > > Thanks, > Kai >
