Siyao Meng created HDDS-17451:
---------------------------------
Summary: SCM terminates when it applies a
transitionDeletingOrDeletedToClosedState Raft log entry written by SCM 2.0 or
2.1, and again on every restart
Key: HDDS-17451
URL: https://issues.apache.org/jira/browse/HDDS-17451
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Affects Versions: 2.2.1, 2.3.0, 2.2.0
Reporter: Siyao Meng
h3. Mechanism
SCM 2.0.x and 2.1.x write {{CONTAINER}} Raft log entries for
{{ContainerStateManager.transitionDeletingOrDeletedToClosedState(ContainerID)}}
(HDDS-11136, HDDS-12421) when the leader gets a non-empty replica report for a
DELETING or DELETED container. HDDS-14684 (2.2.0) renamed the method to
{{transitionDeletingOrDeletedToTargetState(ContainerID, LifeCycleState)}} and
kept no method under the old name, so the generated
{{ContainerStateManagerInvoker.invokeLocal}} has no case for it. Applying such
an entry throws {{IllegalArgumentException("Method not found:
transitionDeletingOrDeletedToClosedState in ContainerStateManager")}}. That is
not an {{SCMException}}, so {{SCMStateMachine.applyTransaction}} calls
{{ExitUtils.terminate}}. The entry is still unapplied, so the SCM replays it
and terminates again on every start. The request encoding is unchanged
(parameter class name plus proto bytes), only the method name lookup fails.
HDDS-16169 fixed the same failure for the renamed
{{ContainerStateManager.updateContainerState}}.
h3. Trigger
An SCM on 2.2.x or master has to apply such an entry, which happens when it did
not apply it before the upgrade:
# An SCM that was down or behind when its ring wrote the entry on 2.0.x or
2.1.x, and is then started on 2.2.x or later. It catches up by log append.
# An SCM stopped uncleanly (crash, kill) after the entry and before a
transaction buffer flush or snapshot covered it, then started on 2.2.x or
later. A graceful stop takes a snapshot first and is not affected.
h3. Impact
The affected SCM cannot start on 2.2.0, 2.2.1 or master. If all SCMs were
stopped uncleanly right after such an entry, no SCM can start. No data loss. A
workaround (argued, not run) is to start the stuck SCM on 2.1.x until it has
applied the entry, then upgrade it again.
h3. Fix
As in HDDS-16169, {{ContainerStateManager}} declares
{{transitionDeletingOrDeletedToClosedState(ContainerID)}} again as
{{@Deprecated}} and {{@Replicate}}, {{ContainerStateManagerImpl}} delegates it
to {{transitionDeletingOrDeletedToTargetState}} with CLOSED (the same checks
and result as 2.1), and {{ContainerStateManagerInvoker}} is regenerated.
Nothing on master calls it. No Raft log or proto change.
{{TestContainerStateManager#testApplyLegacyTransitionDeletingOrDeletedToClosedState}}
encodes the entry as 2.1 does and applies it through the SCM Ratis server stub
for a DELETING and a DELETED container. Without the fix it fails with the
{{Method not found}} error above. With it the container ends CLOSED.
A comparison of the {{@Replicate}} methods in 2.0, 2.1 and 2.2.1 with the
{{invokeLocal}} cases on master finds no other missing method besides the ones
tracked in HDDS-17421 ({{FinalizationStateManager}}) and HDDS-17447
({{DeletedBlockLogStateManager}}).
Found while reviewing the fix for HDDS-17447. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Generated with Claude
Code (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
