Siyao Meng created HDDS-16896:
---------------------------------
Summary: Bootstrapped SCM builds its container safe mode rules
from its empty DB and leaves safe mode before datanodes report its containers
Key: HDDS-16896
URL: https://issues.apache.org/jira/browse/HDDS-16896
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-reload-container-safemode-rules-after-catch-up.patch,
TestBugMC2BootstrappedScmSafeMode.java
h3. Mechanism
{{RatisContainerSafeModeRule}} and {{ECContainerSafeModeRule}} read the CLOSED
and QUASI_CLOSED containers with keys from the container DB once, in the SCM
constructor ({{AbstractContainerSafeModeRule.initializeRule}}). That is before
{{StorageContainerManager.start}} replays the Ratis log or installs a snapshot.
Afterwards the periodic refresh ({{reinitializeRule}}) only removes DELETED
containers, by design since HDDS-15238, and
{{SCMSafeModeManager.refreshAndValidate}} does nothing when the periodic
refresh is enabled (the default).
On a newly bootstrapped SCM the DB is still empty when the rules are built, so
both rules see 0 of 0 containers and {{getCurrentContainerThreshold}} returns
1. Both are marked validated on the next safe mode refresh (every 5s), usually
before the state machine has caught up. Nothing reloads them, so even when
catch-up is faster, they are validated at 0 of 0. A validated rule is never
revalidated, so as soon as the datanode, pipeline and state machine rules pass,
the SCM leaves safe mode, however few of the containers in the installed
snapshot the datanodes have reported.
h3. Trigger
# An HA cluster with CLOSED containers that have keys.
# A new SCM is bootstrapped and catches up from the leader (in every run, by
installing a checkpoint of the leader DB, with the default Ratis settings).
# Only some datanodes report to the new SCM, as during a rolling update of
{{ozone.scm.nodes}} on the datanodes, or when datanodes register unevenly.
# The new SCM leaves safe mode once the minimum number of datanodes and the
pipeline rules are satisfied.
h3. Impact
* The new SCM leaves safe mode with only part of its closed containers
reported. In the reproduction this was 6 of 10 containers, with a cutoff of 99%.
* If it becomes leader before the remaining datanodes report to it (leadership
transfer, or failover during the rollout), ReplicationManager runs on that
partial view after {{hdds.scm.wait.time.after.safemode.exit}} (5 minutes by
default, 3 seconds in the mini cluster). In the reproduction it classified the
4 unreported single replica containers as MISSING. MISSING is only a
classification: no command is sent for it.
* Containers with at least one reported replica are handled as on any SCM that
leaves safe mode before all datanodes have reported. What this defect adds is
that containers with no reported replica no longer keep the SCM in safe mode,
so the container check of safe mode does nothing on a bootstrapped SCM. I found
no path to data loss.
* The effect ends when the remaining datanodes register with the new SCM.
h3. Reproduction
PASS on the unmodified source. [^TestBugMC2BootstrappedScmSafeMode.java]
({{hadoop-ozone/integration-test}}) runs a mini cluster with 3 SCMs, 1 OM and 5
datanodes, and uses the client, {{closeContainer}}, {{bootstrapSCM}},
{{transferLeadership}} and the datanode reconfiguration handler, as an operator
adding an SCM would. It writes RATIS/ONE keys, closes their containers,
bootstraps a fourth SCM with the default Ratis settings and points only the
three datanodes of one RATIS/THREE pipeline at it. Before any datanode
registers, the new SCM reports {{RatisContainerSafeModeRule validate=true: ...
Containers(0 / 0)}} although it holds 10 closed containers with keys. After the
three datanodes register, it leaves safe mode within about 1s with 6 of 10
containers reported. After leadership is transferred to it, ReplicationManager
runs and reports MISSING=4. A passing test means the defect is present.
With the patch the same test fails at its first assertion. The new SCM logs
"Initialized RATIS Containers threshold count to 10" once it has caught up,
stays in safe mode at {{Containers(6 / 10)}}, and ReplicationManager does not
run after the leadership transfer.
Not run: the restart variant (a restarted SCM whose replayed log closes
containers). Containers closed while that SCM was down are replayed with the
key count it had persisted, which is usually 0 for a container that was open
when it stopped, so the rule mostly skips them anyway.
h3. Suggested fix
Rebuilding the rules after every snapshot install is not safe. After the
datanode server has started, a rebuild would refill the list from the DB
without resetting the reported count. Containers that registered datanodes have
already reported would be back in the list and could be counted a second time.
Containers closed since registration would raise the total, but datanodes
report containers to these rules only when they register, so SCM could stay in
safe mode indefinitely (the concern behind HDDS-15238). Clearing an already
validated rule from {{validatedRules}} at readiness would race with the refresh
thread. The patch instead keeps the rules from validating until the data they
count is current.
h3. Patch
[^MC-2-reload-container-safemode-rules-after-catch-up.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
When the SCM has a Ratis state machine, {{SafeModeRuleFactory}} passes it to
the two container rules. Until the state machine is ready,
{{AbstractContainerSafeModeRule.validate}} returns false. On its first call
after that, it reads the containers again with {{initializeRule}}. Datanode
reports only reach the rules after the state machine is ready, because the
datanode server is started in
{{SCMStateMachine.tryStartDNServerAndRefreshSafeMode}}, and each report goes
through {{validate}} before it is processed. So the reload always runs before
any container is counted, and no report is lost. Rules without a state machine
({{SCMHAManagerStub}}, unit tests) behave as before. An SCM whose DB is still
empty after catch-up still validates the rules at once.
The fix relies on the datanode server starting only after the state machine is
ready, which came with HDDS-14989 (2.3.0). Released versions (2.2.0 and 2.2.1)
build the rules the same way and start the datanode server in
{{StorageContainerManager.start}} unconditionally, so they are also affected
(argued from the code, not run). They need HDDS-14989 together with this fix:
without it, reports processed before readiness stay counted, because
{{initializeRule}} does not reset the reported count or the EC replica map, so
the reload could over count.
Covered by {{testContainerRulesCountContainersLoadedBeforeStateMachineReady}}
in the existing {{TestSCMSafeModeManager}}. The rules are built from an empty
DB, 10 closed containers are then added and the state machine becomes ready,
and a datanode reports half of them. Without the change SCM leaves safe mode at
that point. With it, SCM stays in safe mode until the remaining containers are
reported, and with 0 containers it leaves safe mode on the first registration
as before. With the patch, the safe mode unit suites (76 tests) and the mini
cluster suites {{TestSCMFollowerCatchupWithContainerReport}},
{{TestDatanodeSCMNodesReconfiguration}}, {{TestSafeModeCheckSubcommandHA}},
{{TestSCMInstallSnapshot}}, {{TestStorageContainerManagerHA}},
{{TestSCMSnapshot}} and {{TestFailoverWithSCMHA}} pass, and checkstyle is clean.
Not changed: the same cause makes {{OneReplicaPipelineSafeModeRule}} pass with
a threshold of 0 on a bootstrapped SCM. It is built from the empty pipeline DB,
validated on the first refresh, and its refresh only rebuilds it while it is
not validated, so the threshold is never corrected. What is lost is the check
that 90% of the open RATIS/THREE pipelines have a reported datanode before
exit. Its effect is limited, because {{HealthyPipelineSafeModeRule}} is rebuilt
on every refresh until it passes and still waits for at least one healthy
pipeline (with the default {{hdds.scm.safemode.min.datanode}} of 3). It is left
for a follow up.
Found by TLA+ model checking and code review of the SCM safe mode rules under
HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing. The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
