Background: In KIP-405: Kafka Tiered Storage - Apache Kafka - Apache Software Foundation, kafka introduced the feature of hierarchical storage, and RLMM is responsible for storing remote segment's metadata.
BTW, [KAFKA-9555] Topic-based implementation for the RemoteLogMetadataManager - ASF JIRA (apache.org) implements the default RLMM - 'TopicBased-RLMM'. Problem: TopicBased RLMM stores all metadata of subscriptions in memory. In our practice, we found that when the metadata gradually increases, there will be a huge burden on the broker's memory (GB level), and at the same time, it will be very time-consuming to save the snapshot of the full amount of metadata to the disk. Solution We hope to introduce rocksdb to solve this problem: - Implement a RocksdbBasedMetadataCache - All metadata is stored on disk, only a small amount of rocksdb memory cache is required. - There is no need to bear the time consumption caused by saving the full amount of snapshot metadata to disk, rocksdb can guarantee incremental storage. You are welcome to discuss this Improvement by replying email ! Thanks, Hzh0425 | | hzhkafka | | hzhka...@163.com |
