Siyao Meng created HDDS-17151:
---------------------------------
Summary: OM fails to start after downgrade because db.snapdiff is
not pinned to RocksDB format_version 5
Key: HDDS-17151
URL: https://issues.apache.org/jira/browse/HDDS-17151
Project: Apache Ozone
Issue Type: Bug
Components: Snapshot
Reporter: Siyao Meng
Assignee: Siyao Meng
h3. Problem
After a downgrade before finalization, an OM that has run a snapshot diff fails
to start:
{code}
ERROR om.OzoneManagerStarter: Cancelling prepare to start OM in upgrade mode
failed with exception
java.lang.RuntimeException: org.rocksdb.RocksDBException: Corrupt or
unsupported format_version: 7 in file /data/metadata/db.snapdiff/MANIFEST-000037
{code}
h3. Root cause
HDDS-14225 pinned {{BlockBasedTableOptions.format_version}} to 5
({{ManagedBlockBasedTableConfig.FORMAT_VERSION}}) in {{DBProfile}} and
{{RDBSstFileWriter}}. This keeps SST files readable by RocksDB 7.7.3 (Ozone
2.2.x) until finalization.
{{OmSnapshotManager}} opens the snapshot diff DB with a plain
{{ManagedColumnFamilyOptions}} and bypasses {{DBProfile}}. That DB is
{{db.snapdiff}}, holding snap-diff-job-table, snap-diff-report-table and
snap-diff-purged-job-table. As a result it gets RocksDB's default
{{format_version}}:
* 6 with rocksdbjni 10.10.1.1 (current master)
* 7 with 11.8.1 (HDDS-17143)
RocksDB < 8.6 can read neither. Only an OM that ran a snapshot diff job, i.e.
the leader at that time, is affected.
All other RocksDB writers were checked:
* OM, SCM and datanode DBs use {{DBProfile}}.
* Snapshot defrag and SST pruning use {{RDBSstFileWriter}}.
* Datanode container import re-writes rows through a batch.
* The rest only read.
{{db.snapdiff}} is the only DB that bypasses the pin.
h3. Fix
Set a {{ManagedBlockBasedTableConfig}} with {{FORMAT_VERSION}} on the
{{db.snapdiff}} column family options. The default CF, existing CFs and newly
created CFs are all derived from those options, and the table config is closed
by the existing {{ManagedColumnFamilyOptions.closeDeeply}}.
h3. Testing done
* *Found* by a one-time downgrade test, following up on the HDDS-15937 gaps, in
the {{upgrade/compose/ha}} cluster (3 OM, 3 SCM, 5 datanodes, Recon, secure).
** Sequence: 2.2.1 -> master + HDDS-17143 (RocksDB 11.8.1), pre-finalized ->
2.2.1 -> re-upgrade and finalize.
** A snapshot diff ran on 11.8.1. After the downgrade to 2.2.1, the OM that ran
it (om3) failed to start with the error above.
** om3's {{db.snapdiff}} OPTIONS showed {{format_version=7}} and
{{compression=kLZ4Compression}}. Its SSTs reported format_version 7. om1/om2
started fine.
** om3 started again after re-upgrading to 11.8.1.
* *Standalone repro on master:* a DB written with default column family options
by rocksdbjni 10.10.1.1 fails in 7.7.3 with {{Corrupt or unsupported
format_version: 6}}. The same DB written by 11.8.1 fails with {{...
format_version: 7}}.
* *New unit test* {{TestOmSnapshotManager#testSnapshotDiffDbFormatVersion}}
asserts every {{format_version}} in the latest {{db.snapdiff}} OPTIONS file
equals {{FORMAT_VERSION}}.
** It fails without the fix: {{["format_version=6", "format_version=6",
"format_version=6", "format_version=6"]}}.
** With the fix, it and the rest of {{TestOmSnapshotManager}} pass (18/18).
Checkstyle is clean.
h3. Notes
* A {{db.snapdiff}} written before the fix keeps its format 6/7 files until
they are rewritten.
* Not verified: an affected OM should start after moving {{db.snapdiff}} aside.
It only holds snapshot diff job state and reports, so the diffs would have to
be re-run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
