[
https://issues.apache.org/jira/browse/HDDS-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132645#comment-18132645
]
Ethan Rose commented on HDDS-16862:
-----------------------------------
I'm closing this one as invalid. The existing requirement is that all OMs must
be on the same software version for finalization to be valid. In that scenario
two OMs that complete finalization successfully cannot land on different
versions. Finalize is a single Ratis transaction that writes the [last apparent
version|https://github.com/apache/ozone/blob/c46d5edf657e1838b581d587d8ea3b94678ad4f4/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/upgrade/OMCompleteFinalizeUpgradeRequest.java#L72]
after finalization completes to the DB as part of the request. It is not
possible for it to write an intermediate version when finalization fails
partway through. Even if an upgrade action fails, the OM will crash and need to
re-apply everything, including a successful finalization, before it can become
leader and hand out snapshots.
> Finalization from a Ratis snapshot should stop at the version recorded in the
> snapshot (OM, SCM)
> ------------------------------------------------------------------------------------------------
>
> Key: HDDS-16862
> URL: https://issues.apache.org/jira/browse/HDDS-16862
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-4-bound-snapshot-finalization-to-snapshot-version.patch
>
>
> h3. Mechanism
> When an OM installs a Ratis snapshot, {{OzoneManager.instantiateServices}}
> calls {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}. SCM calls
> the same method from {{FinalizationStateManagerImpl.reinitialize}}. If the
> {{#LAYOUTVERSION}} in the installed DB is ahead of the local apparent
> version, the method logs "Attempting to finalize to that version" and then
> calls {{ComponentVersionManager.finalizeUpgrade()}}. That call does not stop
> at the snapshot's version. It follows {{nextVersion()}} up to the local
> software version, runs the upgrade action of every version on the way, and
> the method then writes the local software version into the installed DB.
> So whenever the snapshot records a version below the local software version,
> the node finalizes further than the leader it took the snapshot from. The pre
> ZDU code added by HDDS-4908 behaved the same way: it finalized to the full
> software layout version and only checked afterwards that the result was not
> below the DB version.
> h3. Trigger
> # An OM (or SCM) is down while its peers finalize to some version V.
> # The cluster is then upgraded to a release whose software version is above V
> and is not finalized yet. The down node is started on that release.
> # It installs a snapshot from the leader whose DB records V. Instead of
> stopping at V, it finalizes to its own software version.
> This works with the current version catalogs. For OM, V can be any pre ZDU
> layout feature, and the node then finalizes to {{ZDU (100)}}.
> h3. Impact
> Downgrade after finalization is not supported by design. The problem is that
> one node ends up finalized further than its peers:
> * Its VERSION file and {{#LAYOUTVERSION}} are ahead of its peers, and nothing
> lowers them again. It has run the upgrade actions of versions its peers have
> not finalized, for OM including the {{ZDU}} action.
> * That OM alone reports the newer version to clients and allows the features
> gated on it. While it is leader, those features are in use although the other
> OMs have not finalized them (argued, not run).
> * The newer release is not finalized on its peers, so the design still allows
> downgrading it, but this node holds a version the older release does not know
> and cannot start on it (argued, not run).
> h3. Patch
> [^MC-4-bound-snapshot-finalization-to-snapshot-version.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{ComponentVersionManager}} gets {{finalizeUpgrade(ComponentVersion
> lastVersion)}}, which runs the existing loop and stops after {{lastVersion}}.
> {{finalizeUpgrade()}} delegates with the software version, so its behaviour
> is unchanged. {{finalizeFromSnapshotIfRequired}} passes the version read from
> the snapshot DB, so OM and SCM finalize from a snapshot exactly to the
> version the snapshot records, and run only the upgrade actions up to it. A
> snapshot from a fully finalized leader of the same release records the
> software version, so that case behaves as before.
> Regression tests in the patch:
> {{testFinalizeFromSnapshotStopsAtSnapshotVersion}} in the existing
> {{TestOMVersionManager}} and {{TestScmVersionManager}}, using the real
> version catalogs and VERSION files. Each test covers three cases:
> * A snapshot between the node's apparent and software versions: the node
> stops at the snapshot's version and stays pre-finalized. Its upgrade action
> runs and the {{ZDU}} action does not.
> * A snapshot at the software version: the node finalizes to it.
> * A snapshot that is not ahead, or a DB without a version: nothing changes.
> Without the change they fail with "expected: <HBASE_SUPPORT (7)> but was:
> <ZDU (100)>" (OM) and "expected: <ERASURE_CODED_STORAGE_SUPPORT (3)> but was:
> <ZDU (100)>" (SCM). With it, the following pass and checkstyle is clean:
> * {{TestOMVersionManager}}, {{TestScmVersionManager}} and
> {{TestDatanodeVersionManager}}.
> * The OM finalize request and service unit tests.
> * {{TestOMUpgradeFinalization}} and {{TestScmHAFinalization}} on a mini
> cluster, including their snapshot finalization tests.
> Found by TLA+ model checking and code review of the OM upgrade finalization
> paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
