fujian-zfj opened a new issue, #7064: URL: https://github.com/apache/rocketmq/issues/7064
### Before Creating the Enhancement Request - [X] I have confirmed that this should be classified as an enhancement rather than a bug/feature. ### Summary We will introduce kv(rocksdb) storage to solve the possible Full GC caused by large object memory allocation when metadata such as topic is serialized, and the performance degradation caused by random writing of a large number of small ConsumeQueue files in a million-topic scenario ### Motivation we have encountered and been plagued by some probelms in the actual production environment. In some actual production scenarios, millions of topics and subsriptions will be created, and both of them may be deleted and created frequently. In current architecture, both topics and subcriptions are persisted in real time, which means each request of topics and subsciptions updating will trigger the persist interface, and the persistence of such metadata is written in full rather than append-only. In the scenario of millions of topics, frequent persistence generates the large memory object jsonString of the topicConfigTable. When the memory is tight, the large memory object jsonString will be directly allocated to the old generation, resulting in frequent Full GC. In addition, millions of topics will inevitably bring millions of indexed ConsumeQueue small files, a large number of indexed small files will destroy the advantages of rocketmq sequential writing, and the random writing of ConsumeQueue will make the performance drop sharply. ### Describe the Solution You'd Like This proposal mainly optimizes and solves the existing perfermance problems in the million-topic scenario from two levels: 1. Metadata like topic, subscription, consumerOffset implement kv storage 2. ConsumeQueue index file implements kv storage ### Describe Alternatives You've Considered no ### Additional Context No response -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
