[
https://issues.apache.org/jira/browse/HDDS-17144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17144:
------------------------------
Fix Version/s: 2.3.0
Target Version/s: 2.2.2 (was: 2.3.0, 2.2.2)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> ozone repair tools fail to load RocksDB OPTIONS written by a newer RocksDB
> version
> ----------------------------------------------------------------------------------
>
> Key: HDDS-17144
> URL: https://issues.apache.org/jira/browse/HDDS-17144
> Project: Apache Ozone
> Issue Type: Bug
> Components: Tools
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
> Attachments: HDDS-17144.001.patch
>
>
> h3. Problem
> {{ManagedRocksDB#openWithLatestOptions}} calls
> {{OptionsUtil.loadLatestOptions}} with a default {{ManagedConfigOptions}}, so
> {{ignore_unknown_options}} is false. It is used by:
> * {{ozone repair om|scm update-transaction}} ({{TransactionInfoRepair}})
> * {{ozone repair ldb compact}} ({{RocksDBManualCompaction}})
> * {{ozone repair om snapshot chain}} ({{SnapshotChainRepair}})
> When the DB was last opened by a newer RocksDB, these tools fail before
> opening the DB. That happens after a downgrade before finalization, or when
> an older CLI runs against a newer DB. Reproduced:
> ||DB written by||Loaded by||Result||
> |10.10.1.1 (master)|7.7.3 (2.2.x)|{{Could not find option: :
> metadata_write_temperature}}|
> |11.8.1|10.10.1.1|{{Extra option not recognized: async_wal_precreate}}|
> |either|either, with ignore_unknown_options=true|loads and opens|
> This hits exactly when a repair is needed: the downgraded service does not
> start, so it has not yet rewritten the OPTIONS file in the old format.
> h3. Proposal
> Set {{setIgnoreUnknownOptions(true)}} in {{openWithLatestOptions}}. This is
> safe:
> * RocksDB honours the flag only for OPTIONS files written by a newer RocksDB
> version. For same-version or older files, unknown options still fail as
> corruption ({{options_parser.cc}}, {{ConfigOptions::ignore_unknown_options}}
> docs).
> * RocksDB's own {{ldb}} exposes {{--ignore_unknown_options}} for the same
> reason.
> * Whether the data is readable is still enforced independently by the SST and
> MANIFEST readers.
> When the older tool opens the DB, RocksDB writes a new OPTIONS file in the
> older format and keeps the previous one. Ozone services never load the DB's
> OPTIONS file; they build options from {{DBProfile}}, the ini file and the
> config. So nothing the services rely on is lost.
> Keep {{DBConfigFromFile}} strict: those ini files are written by users, and a
> typo should still fail.
> h3. Test
> Rewrite a DB's OPTIONS file with {{rocksdb_version=99.0.0}} plus an extra
> unknown CF option, and check that {{openWithLatestOptions}} succeeds. With
> the original version header, it must still fail.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
