Siyao Meng created HDDS-16890:
---------------------------------
Summary: SCM closePipeline leaves a container allocated during the
close OPEN on the CLOSED pipeline until the pipeline is deleted
Key: HDDS-16890
URL: https://issues.apache.org/jira/browse/HDDS-16890
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-close-containers-allocated-during-pipeline-close.patch,
TestBugMC2ClosePipelineOrphanContainer.java
h3. Mechanism
{{PipelineManagerImpl.closePipeline}} first calls
{{closeContainersForPipeline}}, which reads the pipeline's containers and
FINALIZEs each OPEN one (FINALIZE moves it to CLOSING and removes it from the
pipeline, then a CLOSE_CONTAINER event is fired). Only after that does it take
the pipeline manager write lock and move the pipeline to CLOSED. No lock is
held between the read and the CLOSED state. Allocation holds only the pipeline
manager read lock ({{WritableRatisContainerProvider.getContainer}}, used by
{{BlockManagerImpl.allocateBlock}}, and
{{ContainerManagerImpl.allocateContainer}}, used by the admin
{{allocateContainer}} RPC), and {{PipelineStateMap.addContainerToOpenPipeline}}
rejects only CLOSED pipelines. A container allocated in that window is
therefore added to the pipeline and never FINALIZEd. It stays OPEN on the
CLOSED pipeline, or on a deleted one when the caller deletes right after
closing ({{DeadNodeHandler}}, {{closeStalePipelines}}, the datastream port
close).
HDDS-5973 ordered the close as containers first, then pipeline, so that SCM
never has an OPEN container on a closed pipeline; its review states the goal as
"containers are in closing before pipeline remove/close log in ratis".
Concurrent allocation was not discussed there. HDDS-5655 added the read lock,
which keeps allocation away from the CLOSED transition itself but not from the
window before it.
h3. Trigger
Any pipeline close (pipeline action from a datanode, stale or dead node,
decommission or maintenance start, admin close, scrubber, full EC pipeline)
while an allocation that creates a new container runs on the same pipeline. For
Ratis this is ordinary block allocation. A Ratis pipeline in steady state is at
its open container limit, and each FINALIZE in the first pass removes a
container from the pipeline, so a block allocation that picks this still OPEN
pipeline during the close finds it below the limit and creates a new container
({{ContainerManagerImpl.getMatchingContainer}}). The close itself makes
allocation create the late container. The window lasts as long as the FINALIZE
calls of the first pass. For EC, block allocation adds a container only to a
pipeline it has just created and not yet opened
({{WritableECContainerProvider}}). That pipeline is already registered with the
node manager, so a stale node, dead node or decommission handler that closes it
in that moment leaves the container OPEN on it, and the following
{{openPipeline}} fails with "Closed pipeline can not be opened". That is rare;
otherwise only the admin {{allocateContainer}} RPC hits the EC route.
h3. Impact
* The late container gets no FINALIZE and no close command. Allocation offers
only OPEN pipelines, so no later block goes into it.
* {{OpenContainerHandler}} in {{ReplicationManager}} closes an OPEN container
when {{ReplicationManager.hasHealthyPipeline}} is false or a replica is
reported in another state. {{hasHealthyPipeline}} only checks that the pipeline
exists, so a late container whose replicas are OPEN (or that has none yet) is
closed only after the scrubber deletes the CLOSED pipeline. With the default
values ({{ozone.scm.pipeline.destroy.timeout}} 66s,
{{ozone.scm.pipeline.scrub.interval}} 5m,
{{hdds.scm.replication.thread.interval}} 300s) that is up to about 11 minutes
after the close. For callers that delete right away it is the next
ReplicationManager run. If ReplicationManager or the scrubber is stopped, the
container stays OPEN.
* When the close command finally arrives the Ratis group is gone, so the
datanode quasi closes a Ratis container ({{CloseContainerCommandHandler}}, no
write channel and no force flag) instead of closing it through Ratis, and
ReplicationManager has to resolve the quasi closed replicas later.
* What the writer sees: {{BlockManagerImpl.newBlock}} reads the pipeline again
after the container was chosen, so {{allocateBlock}} can return the CLOSED
pipeline with a block in a container SCM still has as OPEN. After a close
without delete the Ratis group stays up until the scrubber deletes the
pipeline, so writes of that block into the OPEN container go to a live group
and succeed. After a close with delete the group is removed, the write fails
and the client excludes the pipeline and asks for a new block, as it does for
the other containers of that pipeline. No path that acknowledges a write on a
removed group was found.
The ReplicationManager, scrubber, datanode and client steps above are argued
from code. The reproduction stops after {{closePipeline}} returns.
h3. Reproduction
[^TestBugMC2ClosePipelineOrphanContainer.java] ({{hadoop-hdds/server-scm}},
target path and Maven command in the class comment) runs a real
{{PipelineManagerImpl}} and {{ContainerManagerImpl}} on RocksDB with
{{MockNodeManager}} and two real threads. A latch in the container manager that
the closer uses holds {{closePipeline}} just before it FINALIZEs the last
container it read, which only fixes the timing. The test thread then allocates
through {{WritableRatisContainerProvider.getContainer}} (Ratis) or
{{ContainerManager.allocateContainer}} (EC 3+2, as the admin RPC does). Output
on unmodified source, the same for both routes:
{noformat}
close race: closePipeline read [#1], pipeline state while it runs = OPEN
close race: after closePipeline: #1 (read by closePipeline) state = CLOSING
close race: allocation during close handed out #2 on Pipeline-...
close race: after closePipeline: pipeline state = CLOSED, containers still in
pipeline = [#2], #2 state = OPEN
close race: OPEN pipelines offered to new allocations = []
{noformat}
No CLOSE_CONTAINER event was fired for #2. A passing test means the defect is
present.
h3. Suggested fix
The patch removes the cause. {{ReplicationManager.hasHealthyPipeline}} could in
addition treat a CLOSED pipeline (not a DORMANT one, whose containers are meant
to stay OPEN) as unhealthy, so that any OPEN container left on a CLOSED
pipeline is closed on the next ReplicationManager run while its Ratis group
still exists. With the patch the only remaining way I found to get there is an
SCM crash or leader change between the CLOSED transition and the second
container pass (argued), so this is not part of the patch.
h3. Patch
[^MC-2-close-containers-allocated-during-pipeline-close.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
{{closePipeline}} calls {{closeContainersForPipeline}} a second time after the
pipeline is CLOSED. FINALIZE removes a container from its pipeline, so the
second call only finds containers added while the first one ran, and normally
reads an empty set. {{addContainerToOpenPipeline}} checks for CLOSED under the
same state manager lock as the CLOSED transition, so once the pipeline is
CLOSED no container can be added and the second call finds every late one. The
allocation path is unchanged and takes no new lock, and the HDDS-5973 order is
kept for every container that existed when the close started. Holding the write
lock from the read to the CLOSED transition would also work but blocks all
allocation while the FINALIZE calls go through Ratis; moving the pipeline to a
new non OPEN state before the read would change the pipeline state machine. The
second call relies on CLOSED being final. At this commit {{activatePipeline}}
can move a CLOSED pipeline back to OPEN, which is reported and fixed separately
in HDDS-16889.
Covered by {{testClosePipelineClosesContainerAllocatedDuringClose}} in the
existing {{TestPipelineManagerImpl}}, which allocates a container from inside
the first FINALIZE and checks that both containers are CLOSING, none is left in
the pipeline and each got exactly one CLOSE_CONTAINER event. Without the change
it fails with "expected: CLOSING but was: OPEN". With it the SCM pipeline,
block, container manager, close container, dead node and client protocol suites
pass (230 tests), as do {{TestPipelineClose}} on a mini cluster (5 tests) and
the Recon pipeline manager tests (5 tests), and checkstyle is clean. With the
patch both tests of the reproduction fail, because the late container is
CLOSING.
Found by TLA+ model checking and code review of the SCM pipeline lifecycle
paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. Related but not duplicates: HDDS-5973 (resolved, close order for
SCM restarts), HDDS-5655 (resolved, read lock around allocation), HDDS-7829
(resolved, rejects adding a container to a CLOSED pipeline), HDDS-10231
(resolved, ReplicationManager closes OPEN containers whose pipeline is missing)
and HDDS-11521 (Not A Problem, client wait on pipeline creation). The attached
patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
