Hi Alexis, Sorry for the late response. I come from the reply in FLINK-27504[1]. The MAINFEST file in RocksDB records history of version changes.
In other words, once a new SST file created or an old file deleted via compaction, it will create a new version in RocksDB, which will update the MAINFEST file. The default value for max MAINFEST file size is 1GB [2], since you create the checkpoint every 30 seconds, files might be flushed on that time, and that's why the MAINFEST file grows. You can limit the max file size via DBOptions#setMaxManifestFileSize [3]. [1] https://issues.apache.org/jira/browse/FLINK-27504?focusedCommentId=17537788&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17537788 [2] https://github.com/ververica/frocksdb/blob/8608d75d85f8e1b3b64b73a4fb6d19baec61ba5c/include/rocksdb/options.h#L636 [3] https://github.com/ververica/frocksdb/blob/8608d75d85f8e1b3b64b73a4fb6d19baec61ba5c/java/src/main/java/org/rocksdb/DBOptions.java#L520 Best Yun Tang ________________________________ From: Alexis Sarda-Espinosa <alexis.sarda-espin...@microfocus.com> Sent: Tuesday, May 3, 2022 8:47 To: Peter Brucia <pe...@ververica.com> Cc: user@flink.apache.org <user@flink.apache.org> Subject: RE: RocksDB's state size discrepancy with what's seen with state processor API Ok Regards, Alexis. From: Peter Brucia <pe...@ververica.com> Sent: Freitag, 22. April 2022 15:31 To: Alexis Sarda-Espinosa <alexis.sarda-espin...@microfocus.com> Subject: Re: RocksDB's state size discrepancy with what's seen with state processor API No Sent from my iPhone
