Hi Kai: Good ideas, For this PIP, there are the following confusions:
1. Does the current bundle-data strongly depend on zk for data consistency? 2. If the data of bundle-data is dumped to system-topic, can there be a good guarantee for the real-time performance of bundle changes? 3. This seems to be an incompatible change, is there a better upgrade plan for current users? -- Thanks Xiaolong Ran Haiting Jiang <jianghait...@apache.org> 于2022年4月7日周四 15:36写道： > Hi Kai, > > Good feature. Please add more info about how to enable this in an existing > cluster. Do we plan to write bundle data to both zk and system topic > during rolling update? > > Thanks, > Haiting > > On 2022/04/07 01:35:09 Kai Wang wrote: > > Hi Pulsar community, > > > > I created a PIP to use the system topic to store the bundle load data in > the load manager. > > > > The proposal can be found: https://github.com/apache/pulsar/issues/15037 > > > > --------- > > ## Motivation > > > > Currently, Pulsar LoadManager is using Zookeeper to store the bundle > load data, when we have many bundles, this might put a lot of pressure on > Zookeeper. > > > > ## Goal > > > > This PIP proposes storing the load manager's bundle load data to a > system topic and using TableView to read it. > > > > > > > > ## Implementation > > > > Since the bundle load data is stats data, it no needs a strong > consistent guarantee. So we can use the system topic to store the load data > and use TableView to read it. > > > > > > ### System topic client > > > > > > > > Add a new SystemTopicClient calls > `LoadBalanceBundleDataSystemTopicClient` , the topic name is > `persistent://pulsar/system/__load_balance_bundle_data`, and the key is the > bundle name, value is BundleData. > > > > ```java > > public class LoadBalanceBundleDataSystemTopicClient extends > SystemTopicClientBase<LoadBalanceBundleDataEvent> { > > // ... > > } > > ``` > > > > > > > > Add new Event calls `LoadBalanceBundleDataEvent` > > > > ``` > > @Data > > public class LoadBalanceBundleDataEvent { > > > > private String bundle; > > > > private BundleData bundleData; > > } > > > > ``` > > > > > > ### ModularLoadManagerImpl > > > > > > Add a TableView in `ModularLoadManagerImpl` to replace the bundle cache > > > > ``` > > private TableView<String, BundleData> bundlesTableView; > > ``` > > > > > > ## Compatibility > > > > This feature can have both backward and forward compatibility since the > bundle data is stats data. > > > > --------- > > > > Thanks, > > Kai > > >
