[
https://issues.apache.org/jira/browse/HDDS-17447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17447:
------------------------------
Status: Patch Available (was: Open)
> SCM terminates when it applies a deleted block retry count Raft log entry
> written by SCM 2.0 or older, and again on every restart
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17447
> URL: https://issues.apache.org/jira/browse/HDDS-17447
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.3.0, 2.2.1
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Attachments:
> CR-11-apply-legacy-deleted-block-retry-count-log-entries.patch,
> TestBugCR11LegacyRetryCountEntryReplay.java
>
>
> h3. Mechanism
> SCM 1.3.0 to 2.0.x write {{BLOCK}} Raft log entries for
> {{DeletedBlockLogStateManager.increaseRetryCountOfTransactionInDB}} (when a
> delete transaction goes over {{ozone.scm.block.deletion.max.retry}}, default
> 4096) and {{resetRetryCountOfTransactionInDB}} (on {{ozone admin scm
> deletedBlocksTxn reset}}). 2.1.0 stopped writing them (HDDS-13520) and kept
> both methods as no-ops so that it can still apply them. HDDS-14915 kept them
> for that compatibility and HDDS-14916 (2.2.0) made them {{default}} methods
> of the interface, but the generated
> {{DeletedBlockLogStateManagerInvoker.invokeLocal}} has no case for default
> methods. Applying either entry throws {{IllegalArgumentException("Method not
> found: increaseRetryCountOfTransactionInDB in
> DeletedBlockLogStateManager")}}. That is not an {{SCMException}}, so
> {{SCMStateMachine.applyTransaction}} calls {{ExitUtils.terminate}}. The entry
> is still unapplied, so the SCM replays it and terminates again on every
> start. The request encoding is unchanged since 2.0.0 (checked byte for byte
> in the reproduction), only the method name lookup fails. HDDS-17421 is the
> same failure for the pre ZDU finalization entries, and HDDS-16169 fixed it
> for the renamed {{ContainerStateManager.updateContainerState}}.
> h3. Trigger
> An SCM on 2.2.x or master has to apply such an entry, which happens when it
> did not apply it before the upgrade:
> # An SCM that was down or behind when its ring wrote the entry on 1.3.x to
> 2.0.x, and is then started on 2.2.x or later. It catches up by log append.
> # An SCM stopped uncleanly (crash, kill) after the entry and before a
> transaction buffer flush or snapshot covered it, then started on 2.2.x or
> later. A graceful stop takes a snapshot first and is not affected.
> An upgrade through 2.1.x is not affected, because 2.1.x applies both entries.
> h3. Impact
> The affected SCM cannot start on 2.2.0, 2.2.1 or master. If all SCMs were
> stopped uncleanly right after such an entry, no SCM can start. No data loss.
> A workaround (argued, not run) is to start the stuck SCM on its old release
> or on 2.1.x until it has applied the entry, then upgrade it again.
> h3. Reproduction
> PASS on unmodified source. [^TestBugCR11LegacyRetryCountEntryReplay.java]
> ({{hadoop-hdds/server-scm}}) starts one real SCM with its own single node
> Ratis ring, checks that the entry it builds is byte for byte the one SCM
> 2.0.0 writes, appends it through the SCM's Ratis server, and restarts the SCM
> once. For both methods, the SCM terminates with {{Method not found: <method>
> in DeletedBlockLogStateManager}} on apply and again after the restart. A
> passing test means the defect is present. Not run against a real 2.0.x ring.
> h3. Patch
> [^CR-11-apply-legacy-deleted-block-retry-count-log-entries.patch], against
> master d5397d675e6. It also applies to ozone-2.2 at 7dbea64856d, and together
> with the HDDS-17421 patch.
> As in HDDS-16169, both methods are abstract again in
> {{DeletedBlockLogStateManager}}, still {{@Deprecated}} and {{@Replicate}},
> {{DeletedBlockLogStateManagerImpl}} implements them as no-ops, and
> {{DeletedBlockLogStateManagerInvoker}} is regenerated.
> {{resetRetryCountOfTransactionInDB}} now returns {{void}}: with a case for it
> but the old {{int}} return type, applying the entry still terminates the SCM,
> with {{Failed to resolve int}}, because {{SCMRatisResponse.encode}} cannot
> encode a primitive return type. The old return value has no caller. No Raft
> log or proto change.
> {{TestScmInvokerCodeGenerator}} now also checks, for every SCM handler
> interface, that each {{@Replicate}} method has a case in the generated
> {{invokeLocal}} and does not return a primitive. Without the change it fails
> for {{DeletedBlockLogStateManager}}, and it would catch the next method made
> {{default}} the same way. With it, {{TestScmInvokerCodeGenerator}},
> {{TestDeletedBlockLog}}, {{TestSCMStateMachine}} and
> {{TestSCMHATransactionBufferMonitorTask}} pass, and checkstyle is clean. With
> the patch the reproduction fails at its first assertion, because both entries
> are applied.
> Found while validating the results of HDDS-17231, part of the TLA+
> verification effort under HDDS-15926, on commit d5397d675e6. Checked against
> HDDS issues and apache/ozone pull requests for duplicates before filing. The
> attached patch is a proposal for review. Generated with Claude Code (Claude
> Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
