Siyao Meng created HDDS-17089:
---------------------------------
Summary: SCM that stops during snapshot install after the DB swap
restarts pre-finalized and shuts down every finalized datanode
Key: HDDS-17089
URL: https://issues.apache.org/jira/browse/HDDS-17089
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-3-finalize-scm-from-db-on-startup.patch,
TestBugMC3PreFinalizedScmAfterInterruptedInstall.java
h3. Mechanism
The SCM keeps its apparent version in two places: the VERSION file
({{SCMStorageConfig}}) and {{#LAYOUTVERSION}} in the SCM DB meta table. On
startup {{ScmVersionManager}} is built from the VERSION file only, and nothing
on the SCM startup path reads {{#LAYOUTVERSION}}.
{{SCMHAManagerImpl.installCheckpoint}} first replaces {{scm.db}} with the
leader's checkpoint ({{HAUtils.replaceDBWithCheckpoint}}) and only then, at the
end of {{startServices}}, {{FinalizationStateManagerImpl.reinitialize}} calls
{{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}, which writes the
VERSION file. If the SCM process is killed in between, the DB is finalized and
the VERSION file is not. The window runs from the end of the checkpoint copy
(the transient DB marker is gone, so the DB opens normally on restart) through
opening the new DB and reloading the pipeline, container and deleted block
state, to the first VERSION write. A reload failure in that window restores the
DB backup, so only a crash (kill, OOM, host failure) leaves this state.
After a restart nothing repairs it:
* The SCM constructor does not compare the VERSION file with the DB.
* The Raft log is replayed from the DB's transaction index, which is the
checkpoint index, so the FINALIZE entry below it is never applied on this SCM.
h3. Trigger
# {{ozone.scm.ha.ratis.log.purge.enabled}} is true. With the default (false)
Ratis does not purge past a peer's commit index, so a down or lagging SCM
catches up from the log and does not install a snapshot (from code, not run). A
newly added SCM installs a snapshot at any setting, but a fresh SCM starts at
its software version, so it does not finalize from it.
# An SCM does not apply the finalization before the leader purges its log past
it: it stops or falls behind after the finalize command passed the peer version
check, or finalize was forced while it was down. A plain finalize is refused
while a peer SCM is unreachable.
# The SCM starts and installs a snapshot from the leader. The checkpoint
carries the finalized version.
# The SCM process is killed after the DB swap and before the VERSION file is
written.
# It is restarted and catches up from the Raft log without another snapshot
install.
h3. Impact
* The SCM runs pre-finalized on top of a finalized DB, while its peers are
finalized (run).
* Once it has caught up and starts its datanode server,
{{SCMNodeManager.shouldFenceDatanode}} rejects every finalized datanode that
registers with it, and {{RegisterEndpointTask}} shuts each of them down. All
datanodes register with a restarted SCM, so all of them stop (run with three
datanodes, all three stopped). A datanode restarted later stops again while the
SCM stays pre-finalized (from code, not run).
* It stays that way until a later snapshot install on that SCM, or until
finalize is run again, which this SCM then applies, and the stopped datanodes
are restarted (from code, not run).
* No data loss.
h3. Reproduction
PASS on the unmodified commit, about 41 s.
[^TestBugMC3PreFinalizedScmAfterInterruptedInstall.java]
({{hadoop-ozone/integration-test}}) uses a three SCM HA mini cluster set up as
in {{TestScmHAFinalization.testSnapshotFinalization}}: one SCM is initialized
but joins the ring only after the other two are finalized (the inactive SCM of
that test), with its VERSION file at {{INITIAL_VERSION}} through the testing
key {{testing.hdds.scm.init.apparent.version}}, so it installs the leader's
snapshot as a bootstrapping peer. The test finalizes the two running SCMs
(forced, since the third is not running) and the datanodes and advances the
log. Before the third SCM starts, a test upgrade action is registered (through
{{ScmVersionManager.getUpgradeActionsForTesting}}) on the first version its
snapshot finalization walks. That action runs after the DB swap and before the
first VERSION write; it copies the SCM metadata directory, which is what a kill
leaves on disk, and throws to stop the install. The test stops that SCM,
restores the copy and starts a new SCM on it. No real process kill was run.
Result: {{Restarted SCM: VERSION file 0, apparent version INITIAL_VERSION (0),
DB version 100}}, then for each datanode "has apparent version ZDU (100) which
is higher than SCM apparent version INITIAL_VERSION (0). SCM must finalize
before datanodes." and "This datanode will shut down.", and finally all three
datanodes stopped with the SCM caught up at applied index (t:2, i:74), still at
INITIAL_VERSION. One snapshot install on that SCM in total. A passing test
means the defect is present.
h3. Patch
[^MC-3-finalize-scm-from-db-on-startup.patch], against
7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef. It also applies to master at
268654683b4.
On startup, right after {{ScmVersionManager}} is built in
{{StorageContainerManager.initializeSystemManagers}}, the SCM now calls
{{RatisBasedVersionManager.finalizeToDBVersionIfRequired}}. It finalizes only
up to the version recorded in the DB, never to the software version of the
binary, and logs a warning when it does. It is a no-op unless the DB records a
version the VERSION file does not allow; on a normal restart the DB is at or
below the VERSION file, since the FINALIZE apply writes the VERSION file before
the buffered {{#LAYOUTVERSION}}. It runs before the Ratis server and the
datanode server start. The SCM upgrade actions it may run are safe at that
point, and the {{ScmUpgradeAction}} javadoc now notes that an action may run
during SCM construction, before the managers created after the version manager
exist. An SCM whose {{#LAYOUTVERSION}} holds a version the binary does not know
now fails at startup instead of ignoring the DB value, as with HDDS-16863 on OM.
Related: HDDS-16863 (the same gap on OM) and HDDS-16862 (bound finalization
from a snapshot). {{finalizeToDBVersionIfRequired}} and the bounded
{{ComponentVersionManager.finalizeUpgrade(lastVersion)}} are the helpers
proposed for OM in HDDS-16863, with identical hunks (HDDS-16862 adds the same
{{finalizeUpgrade}} hunk). The patches merge cleanly in either order ({{git
merge}}, or {{git apply --3way}} for the patch files); once HDDS-16863 is in,
this patch reduces to the three lines in {{StorageContainerManager}}, the
javadoc note and the test.
Covered by {{testStartupFinalizesToApparentVersionInDB}} in the existing
{{TestScmStartupInvalidApparentVersion}}. On one SCM storage it checks four
cases:
* Without a DB version the SCM stays pre-finalized.
* A DB at {{ERASURE_CODED_STORAGE_SUPPORT}} over a VERSION file at
{{INITIAL_VERSION}} finalizes the SCM to {{ERASURE_CODED_STORAGE_SUPPORT}} and
no further.
* A DB behind the VERSION file changes nothing.
* A DB at the software version finalizes the SCM fully.
Without the change it fails with {{expected: ERASURE_CODED_STORAGE_SUPPORT (3)
but was: INITIAL_VERSION (0)}}. With it, these pass and checkstyle is clean:
* {{TestScmStartupInvalidApparentVersion}} (3 tests)
* the SCM, datanode and OM version manager suites,
{{TestClearFinalizingStateScmUpgradeAction}}, {{TestSCMHAManagerImpl}},
{{TestSCMStateMachine}}, the SCM construction and starter suites,
{{TestDeletedBlockLog}} and {{TestBlockManager}} (145 tests)
* {{TestScmHAFinalization}}, {{TestScmDataDistributionFinalization}} and
{{TestSCMInstallSnapshotWithHA}} (8 tests)
With the patch the reproduction fails, because the restarted SCM finalizes in
its constructor ({{expected: INITIAL_VERSION (0) but was: ZDU (100)}}).
Found by TLA+ model checking and code review of the SCM upgrade finalization
paths under HDDS-15926, on commit 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
