[
https://issues.apache.org/jira/browse/HDDS-13422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-13422:
-------------------------------
Description:
We tried to use the offline RocksDB compaction tool to delete all the
tombstones from the keyTable in OM DB. We used RdbUtil#getLiveSSTFilesForCFs to
check for the LiveFileMetadata for the SST files on keyTable and confirmed that
the numDeletions are all zero.
However, we saw that the compacted OM DB result in larger number of SST files
(14332 -> 16111) and larger size (800GB -> 1.1T). We found out that the OPTIONS
files before compaction and after the compaction are different. Notably the
block_size is reduced from 16KB to 4KB. This might explain why there are larger
number of SST files (due to file splitting) and the size increase (due to lower
compression ratio on the smaller blocks).
One suspect is that RocksDB#open will simply reinitialize the RocksDB options
to the default options. We should use RDBStore instead that will use the
correct RocksDB configuration in DBProfile.
was:
We tried to use the offline RocksDB compaction tool to delete all the
tombstones from the keyTable in OM DB. We used RdbUtil#getLiveSSTFilesForCFs to
check for the LiveFileMetadata for the SST files on keyTable and confirmed that
the numDeletions are all zero.
However, we saw that the compacted OM DB result in larger number of SST files
(14332 -> 16111) and larger size (800GB -> 1.1T). We found out that the OPTIONS
files before compaction and after the compaction are different. Notably the
block_size is reduced from 16KB to 4KB. This might explain why there are larger
number of SST files (due to file splitting) and the size increase (due to lower
compression ratio on the smaller blocks).
The suspect is that RocksDB#open will simply reinitialize the RocksDB options
to the default options. We should use RDBStore instead that will use the
correct RocksDB configuration in DBProfile.
> Offline repair command should preserve previous DBProfile configurations
> ------------------------------------------------------------------------
>
> Key: HDDS-13422
> URL: https://issues.apache.org/jira/browse/HDDS-13422
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We tried to use the offline RocksDB compaction tool to delete all the
> tombstones from the keyTable in OM DB. We used RdbUtil#getLiveSSTFilesForCFs
> to check for the LiveFileMetadata for the SST files on keyTable and confirmed
> that the numDeletions are all zero.
> However, we saw that the compacted OM DB result in larger number of SST files
> (14332 -> 16111) and larger size (800GB -> 1.1T). We found out that the
> OPTIONS files before compaction and after the compaction are different.
> Notably the block_size is reduced from 16KB to 4KB. This might explain why
> there are larger number of SST files (due to file splitting) and the size
> increase (due to lower compression ratio on the smaller blocks).
> One suspect is that RocksDB#open will simply reinitialize the RocksDB options
> to the default options. We should use RDBStore instead that will use the
> correct RocksDB configuration in DBProfile.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
