[
https://issues.apache.org/jira/browse/HDDS-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17451:
------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SCM terminates when it applies a transitionDeletingOrDeletedToClosedState
> Raft log entry written by SCM 2.0 or 2.1, and again on every restart
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17451
> URL: https://issues.apache.org/jira/browse/HDDS-17451
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM
> Affects Versions: 2.2.0, 2.3.0, 2.2.1
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> h3. Mechanism
> SCM 2.0.x and 2.1.x write {{CONTAINER}} Raft log entries for
> {{ContainerStateManager.transitionDeletingOrDeletedToClosedState(ContainerID)}}
> (HDDS-11136, HDDS-12421) when the leader gets a non-empty replica report for
> a DELETING or DELETED container. HDDS-14684 (2.2.0) renamed the method to
> {{transitionDeletingOrDeletedToTargetState(ContainerID, LifeCycleState)}} and
> kept no method under the old name, so the generated
> {{ContainerStateManagerInvoker.invokeLocal}} has no case for it. Applying
> such an entry throws {{IllegalArgumentException("Method not found:
> transitionDeletingOrDeletedToClosedState in ContainerStateManager")}}. That
> is not an {{SCMException}}, so {{SCMStateMachine.applyTransaction}} calls
> {{ExitUtils.terminate}}. The entry is still unapplied, so the SCM replays it
> and terminates again on every start. The request encoding is unchanged
> (parameter class name plus proto bytes), only the method name lookup fails.
> HDDS-16169 fixed the same failure for the renamed
> {{ContainerStateManager.updateContainerState}}.
> h3. Trigger
> An SCM on 2.2.x or master has to apply such an entry, which happens when it
> did not apply it before the upgrade:
> # An SCM that was down or behind when its ring wrote the entry on 2.0.x or
> 2.1.x, and is then started on 2.2.x or later. It catches up by log append.
> # An SCM stopped uncleanly (crash, kill) after the entry and before a
> transaction buffer flush or snapshot covered it, then started on 2.2.x or
> later. A graceful stop takes a snapshot first and is not affected.
> h3. Impact
> The affected SCM cannot start on 2.2.0, 2.2.1 or master. If all SCMs were
> stopped uncleanly right after such an entry, no SCM can start. No data loss.
> A workaround (argued, not run) is to start the stuck SCM on 2.1.x until it
> has applied the entry, then upgrade it again.
> h3. Fix
> As in HDDS-16169, {{ContainerStateManager}} declares
> {{transitionDeletingOrDeletedToClosedState(ContainerID)}} again as
> {{@Deprecated}} and {{@Replicate}}, {{ContainerStateManagerImpl}} delegates
> it to {{transitionDeletingOrDeletedToTargetState}} with CLOSED (the same
> checks and result as 2.1), and {{ContainerStateManagerInvoker}} is
> regenerated. Nothing on master calls it. No Raft log or proto change.
> {{TestContainerStateManager#testApplyLegacyTransitionDeletingOrDeletedToClosedState}}
> encodes the entry as 2.1 does and applies it through the SCM Ratis server
> stub for a DELETING and a DELETED container. Without the fix it fails with
> the {{Method not found}} error above. With it the container ends CLOSED.
> A comparison of the {{@Replicate}} methods in 2.0, 2.1 and 2.2.1 with the
> {{invokeLocal}} cases on master finds no other missing method besides the
> ones tracked in HDDS-17421 ({{FinalizationStateManager}}) and HDDS-17447
> ({{DeletedBlockLogStateManager}}).
> Found while reviewing the fix for HDDS-17447. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Generated with
> Claude Code (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
