[
https://issues.apache.org/jira/browse/HDDS-17151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17151:
------------------------------
Status: Patch Available (was: Open)
> OM fails to start after downgrade because db.snapdiff is not pinned to
> RocksDB format_version 5
> -----------------------------------------------------------------------------------------------
>
> Key: HDDS-17151
> URL: https://issues.apache.org/jira/browse/HDDS-17151
> Project: Apache Ozone
> Issue Type: Bug
> Components: Snapshot
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Blocker
> Labels: pull-request-available
> Attachments: HDDS-17151.001.patch
>
>
> h3. Problem
> After a downgrade before finalization, an OM that has run a snapshot diff
> fails to start:
> {code}
> ERROR om.OzoneManagerStarter: Cancelling prepare to start OM in upgrade mode
> failed with exception
> java.lang.RuntimeException: org.rocksdb.RocksDBException: Corrupt or
> unsupported format_version: 7 in file
> /data/metadata/db.snapdiff/MANIFEST-000037
> {code}
> h3. Root cause
> HDDS-14225 pinned {{BlockBasedTableOptions.format_version}} to 5
> ({{ManagedBlockBasedTableConfig.FORMAT_VERSION}}) in {{DBProfile}} and
> {{RDBSstFileWriter}}. This keeps SST files readable by RocksDB 7.7.3 (Ozone
> 2.2.x) until finalization.
> {{OmSnapshotManager}} opens the snapshot diff DB with a plain
> {{ManagedColumnFamilyOptions}} and bypasses {{DBProfile}}. That DB is
> {{db.snapdiff}}, holding snap-diff-job-table, snap-diff-report-table and
> snap-diff-purged-job-table. As a result it gets RocksDB's default
> {{format_version}}:
> * 6 with rocksdbjni 10.10.1.1 (current master)
> * 7 with 11.8.1 (HDDS-17143)
> RocksDB < 8.6 can read neither. Only an OM that ran a snapshot diff job, i.e.
> the leader at that time, is affected.
> All other RocksDB writers were checked:
> * OM, SCM and datanode DBs use {{DBProfile}}.
> * Snapshot defrag and SST pruning use {{RDBSstFileWriter}}.
> * Datanode container import re-writes rows through a batch.
> * The rest only read.
> {{db.snapdiff}} is the only DB that bypasses the pin.
> h3. Fix
> Set a {{ManagedBlockBasedTableConfig}} with {{FORMAT_VERSION}} on the
> {{db.snapdiff}} column family options. The default CF, existing CFs and newly
> created CFs are all derived from those options, and the table config is
> closed by the existing {{ManagedColumnFamilyOptions.closeDeeply}}.
> h3. Testing done
> * *Found* by a one-time downgrade test, following up on the HDDS-15937 gaps,
> in the {{upgrade/compose/ha}} cluster (3 OM, 3 SCM, 5 datanodes, Recon,
> secure).
> ** Sequence: 2.2.1 -> master + HDDS-17143 (RocksDB 11.8.1), pre-finalized ->
> 2.2.1 -> re-upgrade and finalize.
> ** A snapshot diff ran on 11.8.1. After the downgrade to 2.2.1, the OM that
> ran it (om3) failed to start with the error above.
> ** om3's {{db.snapdiff}} OPTIONS showed {{format_version=7}} and
> {{compression=kLZ4Compression}}. Its SSTs reported format_version 7. om1/om2
> started fine.
> ** om3 started again after re-upgrading to 11.8.1.
> * *Standalone repro on master:* a DB written with default column family
> options by rocksdbjni 10.10.1.1 fails in 7.7.3 with {{Corrupt or unsupported
> format_version: 6}}. The same DB written by 11.8.1 fails with {{...
> format_version: 7}}.
> * *New unit test* {{TestOmSnapshotManager#testSnapshotDiffDbFormatVersion}}
> asserts every {{format_version}} in the latest {{db.snapdiff}} OPTIONS file
> equals {{FORMAT_VERSION}}.
> ** It fails without the fix: {{["format_version=6", "format_version=6",
> "format_version=6", "format_version=6"]}}.
> ** With the fix, it and the rest of {{TestOmSnapshotManager}} pass (18/18).
> Checkstyle is clean.
> h3. Notes
> * A {{db.snapdiff}} written before the fix keeps its format 6/7 files until
> they are rewritten.
> * Not verified: an affected OM should start after moving {{db.snapdiff}}
> aside. It only holds snapshot diff job state and reports, so the diffs would
> have to be re-run.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
