[
https://issues.apache.org/jira/browse/HDDS-16448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16448:
------------------------------
Description:
h3. Finding
Internal invariant violation (a CLOSED container's immutable sequenceId is
silently bumped by a healthy higher-BCSID replica report) that demonstrably
propagates to external effect: RatisContainerReplicaCount reclassifies
quorum-correct QUASI_CLOSED replicas as unhealthy (healthy count 3 to 1), so a
fully-replicated CLOSED container is treated as under-replicated and
MismatchedReplicasHandler strands those replicas. Triggerable Level 0 through
the real ContainerReportHandler.onMessage path with an ordinary DN report; the
bump is monotonic with no downstream self-heal.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-3
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-container-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/015-scm-container-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-container-lifecycle|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
h3. Steps to reproduce
Precondition: a CLOSED Ratis container (RF=3) at sequenceId S, with three
QUASI_CLOSED IN_SERVICE replicas whose block commit sequence id equals S (a
normal quasi-closed-stuck situation). RatisContainerReplicaCount counts a
QUASI_CLOSED replica of a CLOSED container as healthy only when its sequenceId
equals the container sequenceId, so the container is healthy with 3 copies.
Action: a healthy replica on a fourth datanode sends an ordinary container
report for the same container with a block commit sequence id higher than S
(replicas reporting a higher BCSID than SCM expects for an already-closed
container is a real occurrence, see the error log added by the "log an error
before increasing the sequence id of a CLOSED container" change).
Observed wrong outcome: AbstractContainerReportHandler.updateContainerStats
bumps the CLOSED container's sequenceId from S to the reported BCSID. The three
QUASI_CLOSED replicas now have sequenceId != container sequenceId, so
RatisContainerReplicaCount reclassifies them as UNHEALTHY (healthy count drops
3 to 1) and the fully replicated container is treated as under-replicated.
MismatchedReplicasHandler will not close those replicas (it requires sequenceId
equality), so nothing downstream self-heals; the bump is monotonic (max()).
h3. Root cause
In AbstractContainerReportHandler.updateContainerStats
(hadoop-hdds/server-scm/.../AbstractContainerReportHandler.java) the CLOSED
case only logs an error, then the following block unconditionally calls
ContainerInfo.updateSequenceId(bcsid) for any healthy replica whose BCSID
exceeds the container sequenceId, regardless of container state.
ContainerInfo.updateSequenceId (hadoop-hdds/common/.../ContainerInfo.java)
guards only with "assert (isOpen() || QUASI_CLOSED)", which is a no-op in
production (assertions disabled), so the max() bump silently applies to CLOSED
containers even though the field is documented as immutable once closed ("The
sequenceId of a close container cannot change").
h3. Suggested fix
The attached patch [^HDDS-16448.001.patch] gates the sequenceId bump in
updateContainerStats so it only applies while the container can legitimately
advance its sequenceId (OPEN, CLOSING, or QUASI_CLOSED), matching the
precondition the assert in ContainerInfo.updateSequenceId already documents.
CLOSED, DELETING, and DELETED containers keep their settled sequenceId. Tests
are added to TestContainerReportHandler: one asserts a CLOSED container's
sequenceId stays immutable and its healthy QUASI_CLOSED replicas are not
reclassified when a higher-BCSID report arrives, and a negative-side test
asserts a QUASI_CLOSED container still advances its sequenceId (the guard is
not too broad).
Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
test, and review before merge.
was:
h3. Finding
Internal invariant violation (a CLOSED container's immutable sequenceId is
silently bumped by a healthy higher-BCSID replica report) that demonstrably
propagates to external effect: RatisContainerReplicaCount reclassifies
quorum-correct QUASI_CLOSED replicas as unhealthy (healthy count 3 to 1), so a
fully-replicated CLOSED container is treated as under-replicated and
MismatchedReplicasHandler strands those replicas. Triggerable Level 0 through
the real ContainerReportHandler.onMessage path with an ordinary DN report; the
bump is monotonic with no downstream self-heal.
h3. Classification
* Verdict: REPRODUCED
* Severity: High
* Source: Specula TLA+ model checking and confirmation debate, finding MC-3
h3. Reproduce
{noformat}
Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
Specula: v1.1.0 (commit c6aa3dfa)
Target: scm-container-lifecycle
Guidance:
campaigns/ozone-9fbf9ee/targets/015-scm-container-lifecycle/.prompt-extra.md
{noformat}
{code:none}
specula run --agent=claude-code --effort=medium --keep-original
--max-parallel=2 \
--enable-reviews --confirm-debate --tlc-memory-limit=28G --tlc-worker-limit=8
\
"scm-container-lifecycle|apache/ozone|Java|Use the target-specific
.prompt-extra.md"
{code}
Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
specification, counterexample, and confirmation debate live in the Specula run
artifacts.
Generated with Specula (Claude Opus 4.8).
> A CLOSED container's sequenceId is silently bumped by a higher BCSID report,
> stranding quorum correct replicas
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16448
> URL: https://issues.apache.org/jira/browse/HDDS-16448
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16448.001.patch
>
>
> h3. Finding
> Internal invariant violation (a CLOSED container's immutable sequenceId is
> silently bumped by a healthy higher-BCSID replica report) that demonstrably
> propagates to external effect: RatisContainerReplicaCount reclassifies
> quorum-correct QUASI_CLOSED replicas as unhealthy (healthy count 3 to 1), so
> a fully-replicated CLOSED container is treated as under-replicated and
> MismatchedReplicasHandler strands those replicas. Triggerable Level 0 through
> the real ContainerReportHandler.onMessage path with an ordinary DN report;
> the bump is monotonic with no downstream self-heal.
> h3. Classification
> * Verdict: REPRODUCED
> * Severity: High
> * Source: Specula TLA+ model checking and confirmation debate, finding MC-3
> h3. Reproduce
> {noformat}
> Ozone commit: 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb
> Specula: v1.1.0 (commit c6aa3dfa)
> Target: scm-container-lifecycle
> Guidance:
> campaigns/ozone-9fbf9ee/targets/015-scm-container-lifecycle/.prompt-extra.md
> {noformat}
> {code:none}
> specula run --agent=claude-code --effort=medium --keep-original
> --max-parallel=2 \
> --enable-reviews --confirm-debate --tlc-memory-limit=28G
> --tlc-worker-limit=8 \
> "scm-container-lifecycle|apache/ozone|Java|Use the target-specific
> .prompt-extra.md"
> {code}
> Discovered under HDDS-16431 (Specula TLA+ verification effort). The TLA+
> specification, counterexample, and confirmation debate live in the Specula
> run artifacts.
> Generated with Specula (Claude Opus 4.8).
> h3. Steps to reproduce
> Precondition: a CLOSED Ratis container (RF=3) at sequenceId S, with three
> QUASI_CLOSED IN_SERVICE replicas whose block commit sequence id equals S (a
> normal quasi-closed-stuck situation). RatisContainerReplicaCount counts a
> QUASI_CLOSED replica of a CLOSED container as healthy only when its
> sequenceId equals the container sequenceId, so the container is healthy with
> 3 copies.
> Action: a healthy replica on a fourth datanode sends an ordinary container
> report for the same container with a block commit sequence id higher than S
> (replicas reporting a higher BCSID than SCM expects for an already-closed
> container is a real occurrence, see the error log added by the "log an error
> before increasing the sequence id of a CLOSED container" change).
> Observed wrong outcome: AbstractContainerReportHandler.updateContainerStats
> bumps the CLOSED container's sequenceId from S to the reported BCSID. The
> three QUASI_CLOSED replicas now have sequenceId != container sequenceId, so
> RatisContainerReplicaCount reclassifies them as UNHEALTHY (healthy count
> drops 3 to 1) and the fully replicated container is treated as
> under-replicated. MismatchedReplicasHandler will not close those replicas (it
> requires sequenceId equality), so nothing downstream self-heals; the bump is
> monotonic (max()).
> h3. Root cause
> In AbstractContainerReportHandler.updateContainerStats
> (hadoop-hdds/server-scm/.../AbstractContainerReportHandler.java) the CLOSED
> case only logs an error, then the following block unconditionally calls
> ContainerInfo.updateSequenceId(bcsid) for any healthy replica whose BCSID
> exceeds the container sequenceId, regardless of container state.
> ContainerInfo.updateSequenceId (hadoop-hdds/common/.../ContainerInfo.java)
> guards only with "assert (isOpen() || QUASI_CLOSED)", which is a no-op in
> production (assertions disabled), so the max() bump silently applies to
> CLOSED containers even though the field is documented as immutable once
> closed ("The sequenceId of a close container cannot change").
> h3. Suggested fix
> The attached patch [^HDDS-16448.001.patch] gates the sequenceId bump in
> updateContainerStats so it only applies while the container can legitimately
> advance its sequenceId (OPEN, CLOSING, or QUASI_CLOSED), matching the
> precondition the assert in ContainerInfo.updateSequenceId already documents.
> CLOSED, DELETING, and DELETED containers keep their settled sequenceId. Tests
> are added to TestContainerReportHandler: one asserts a CLOSED container's
> sequenceId stays immutable and its healthy QUASI_CLOSED replicas are not
> reclassified when a higher-BCSID report arrives, and a negative-side test
> asserts a QUASI_CLOSED container still advances its sequenceId (the guard is
> not too broad).
> Patch is an AI-drafted proposal (Specula + Claude); pending human build, full
> test, and review before merge.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
