[
https://issues.apache.org/jira/browse/HDDS-17421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17421:
------------------------------
Affects Version/s: 2.3.0
> SCM terminates when it applies a Raft log entry written by the pre-ZDU SCM
> finalization, and again on every restart
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17421
> URL: https://issues.apache.org/jira/browse/HDDS-17421
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-9-apply-pre-zdu-scm-finalization-log-entries.patch,
> TestBugCR9PreZduFinalizeEntryReplay.java
>
>
> h3. Mechanism
> When SCM finalizes on releases 1.3.0 to 2.2.1, the leader writes FINALIZE
> Raft log entries named {{addFinalizingMark}}, {{finalizeLayoutFeature}} (one
> {{Integer}} argument, once per layout feature) and {{removeFinalizingMark}}.
> On master the generated {{FinalizationStateManagerInvoker.invokeLocal}} only
> knows {{finalizeUpgrade}} and {{reinitialize}} (HDDS-14570, HDDS-14672,
> HDDS-15377), so applying one of the old entries throws
> {{IllegalArgumentException("Method not found: ... in
> FinalizationStateManager")}}. That is not an {{SCMException}}, so
> {{SCMStateMachine.applyTransaction}} calls {{ExitUtils.terminate}}. The entry
> is still unapplied, so the SCM replays it and terminates again on every
> start. The request encoding ({{SCMRatisProtocol.proto}}, {{SCMRatisRequest}},
> codecs) is unchanged since 2.2.1, only the method name lookup fails.
> HDDS-16169 fixed the same failure for the renamed
> {{ContainerStateManager.updateContainerState}}.
> h3. Trigger
> An SCM has to apply such an entry on the new code, which happens when it did
> not apply it before the upgrade:
> # An SCM that was down or behind while its ring finalized an upgrade on the
> old release, and is then started on the new release. It catches up by log
> append: with the default {{ozone.scm.ha.ratis.log.purge.enabled}} = false the
> leader does not purge entries that another SCM has not committed yet.
> # An SCM stopped uncleanly (crash, kill) after the old finalization and
> before a transaction buffer flush or snapshot covered it, then started on the
> new release. A graceful stop takes a snapshot first and is not affected.
> HDDS-9769 is the same replay with the old code.
> A freshly bootstrapped SCM is normally not affected: it installs the leader's
> checkpoint before it applies any entry (seen in
> {{TestStorageContainerManagerHA#testBootStrapSCM}} with an unpurged log). It
> replays the whole log only in the fallback of HDDS-16899.
> h3. Impact
> The affected SCM cannot start on the new release. If all SCMs were stopped
> uncleanly right after the old finalization, no SCM can start. No data loss.
> Master only, no release has it. A workaround (argued, not run) is to start
> the stuck SCM on the old release until it has applied the entries, then
> upgrade it again; it was never finalized on the new release, so the old
> release can still run it.
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR9PreZduFinalizeEntryReplay.java]
> ({{hadoop-hdds/server-scm}}) starts one real SCM with its own single node
> Ratis ring at the pre-ZDU layout version {{HBASE_SUPPORT}}, appends a
> {{finalizeLayoutFeature(9)}} entry encoded as 2.2.1 encodes it through the
> SCM's Ratis server, and restarts the SCM once. The SCM terminates with
> {{Method not found: finalizeLayoutFeature in FinalizationStateManager}} on
> apply and again after the restart. A passing test means the defect is
> present. Not run against a real 2.2.x ring.
> h3. Patch
> [^CR-9-apply-pre-zdu-scm-finalization-log-entries.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
> e03ba37c41d.
> As in HDDS-16169, {{FinalizationStateManager}} declares the three old methods
> again with {{@Deprecated}}, and {{FinalizationStateManagerInvoker}} is
> regenerated. Unlike HDDS-16169 they are not {{@Replicate}}, so this version
> cannot write them while {{invokeLocal}} still dispatches them.
> {{addFinalizingMark}} and {{removeFinalizingMark}} do nothing (the mark is
> unused, {{ClearFinalizingStateScmUpgradeAction}} removes a leftover one).
> {{finalizeLayoutFeature}} finalizes up to the given layout version and no
> further, as the SCM that wrote it did, using a new
> {{ComponentVersionManager.finalizeUpgrade(ComponentVersion)}} that stops at
> the target; {{finalizeUpgrade()}} delegates to it with the software version
> and behaves as before. A no-op would leave that SCM below its finalized
> datanodes, which it then fences (as in HDDS-17090); finalizing to the
> software version would finalize it past ZDU without an admin finalize. No
> Raft log or proto change.
> Covered by {{testPreZduFinalizationEntriesFinalizeUpToTheirLayoutVersion}} in
> the existing {{TestSCMStateMachine}}: the old entries applied to a real
> {{ScmVersionManager}} at {{HBASE_SUPPORT}} through the real invoker end at
> {{STORAGE_SPACE_DISTRIBUTION}}, a replayed entry changes nothing, a new
> {{finalizeUpgrade}} entry still finalizes to the software version, and an
> unknown operation or an unknown layout version still terminates. Without the
> change it fails with {{Method not found: addFinalizingMark in
> FinalizationStateManager}}. With it, {{TestSCMStateMachine}},
> {{TestScmInvokerCodeGenerator}}, the SCM, datanode and OM version manager
> tests, the SCM upgrade action and startup tests, {{TestSCMHAManagerImpl}} and
> {{TestScmHAFinalization}} pass, and checkstyle is clean. With the patch the
> reproduction fails at its first assertion, because the entry is applied and
> the SCM finalizes to {{WITNESSED_CONTAINER_DB_PROTO_VALUE}}.
> Found by code review of the zero downtime upgrade version gating in SCM, OM
> and the datanode write path, as part of the TLA+ verification effort under
> HDDS-15926, on commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
