[
https://issues.apache.org/jira/browse/HDDS-17089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17089:
------------------------------
Attachment: TestBugMC3PreFinalizedScmAfterInterruptedInstall.java
> SCM that stops during snapshot install after the DB swap restarts
> pre-finalized and shuts down every finalized datanode
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17089
> URL: https://issues.apache.org/jira/browse/HDDS-17089
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-3-finalize-scm-from-db-on-startup.patch,
> TestBugMC3PreFinalizedScmAfterInterruptedInstall.java
>
>
> h3. Mechanism
> The SCM keeps its apparent version in two places: the VERSION file
> ({{SCMStorageConfig}}) and {{#LAYOUTVERSION}} in the SCM DB meta table. On
> startup {{ScmVersionManager}} is built from the VERSION file only, and
> nothing on the SCM startup path reads {{#LAYOUTVERSION}}.
> {{SCMHAManagerImpl.installCheckpoint}} first replaces {{scm.db}} with the
> leader's checkpoint ({{HAUtils.replaceDBWithCheckpoint}}) and only then, at
> the end of {{startServices}}, {{FinalizationStateManagerImpl.reinitialize}}
> calls {{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}, which
> writes the VERSION file. If the SCM process is killed in between, the DB is
> finalized and the VERSION file is not. The window runs from the end of the
> checkpoint copy (the transient DB marker is gone, so the DB opens normally on
> restart) through opening the new DB and reloading the pipeline, container and
> deleted block state, to the first VERSION write. A reload failure in that
> window restores the DB backup, so only a crash (kill, OOM, host failure)
> leaves this state.
> After a restart nothing repairs it:
> * The SCM constructor does not compare the VERSION file with the DB.
> * The Raft log is replayed from the DB's transaction index, which is the
> checkpoint index, so the FINALIZE entry below it is never applied on this SCM.
> h3. Trigger
> # {{ozone.scm.ha.ratis.log.purge.enabled}} is true. With the default (false)
> Ratis does not purge past a peer's commit index, so a down or lagging SCM
> catches up from the log and does not install a snapshot (from code, not run).
> A newly added SCM installs a snapshot at any setting, but a fresh SCM starts
> at its software version, so it does not finalize from it.
> # An SCM does not apply the finalization before the leader purges its log
> past it: it stops or falls behind after the finalize command passed the peer
> version check, or finalize was forced while it was down. A plain finalize is
> refused while a peer SCM is unreachable.
> # The SCM starts and installs a snapshot from the leader. The checkpoint
> carries the finalized version.
> # The SCM process is killed after the DB swap and before the VERSION file is
> written.
> # It is restarted and catches up from the Raft log without another snapshot
> install.
> h3. Impact
> * The SCM runs pre-finalized on top of a finalized DB, while its peers are
> finalized (run).
> * Once it has caught up and starts its datanode server,
> {{SCMNodeManager.shouldFenceDatanode}} rejects every finalized datanode that
> registers with it, and {{RegisterEndpointTask}} shuts each of them down. All
> datanodes register with a restarted SCM, so all of them stop (run with three
> datanodes, all three stopped). A datanode restarted later stops again while
> the SCM stays pre-finalized (from code, not run).
> * It stays that way until a later snapshot install on that SCM, or until
> finalize is run again, which this SCM then applies, and the stopped datanodes
> are restarted (from code, not run).
> * No data loss.
> h3. Reproduction
> PASS on the unmodified commit, about 41 s.
> [^TestBugMC3PreFinalizedScmAfterInterruptedInstall.java]
> ({{hadoop-ozone/integration-test}}) uses a three SCM HA mini cluster set up
> as in {{TestScmHAFinalization.testSnapshotFinalization}}: one SCM is
> initialized but joins the ring only after the other two are finalized (the
> inactive SCM of that test), with its VERSION file at {{INITIAL_VERSION}}
> through the testing key {{testing.hdds.scm.init.apparent.version}}, so it
> installs the leader's snapshot as a bootstrapping peer. The test finalizes
> the two running SCMs (forced, since the third is not running) and the
> datanodes and advances the log. Before the third SCM starts, a test upgrade
> action is registered (through
> {{ScmVersionManager.getUpgradeActionsForTesting}}) on the first version its
> snapshot finalization walks. That action runs after the DB swap and before
> the first VERSION write; it copies the SCM metadata directory, which is what
> a kill leaves on disk, and throws to stop the install. The test stops that
> SCM, restores the copy and starts a new SCM on it. No real process kill was
> run.
> Result: {{Restarted SCM: VERSION file 0, apparent version INITIAL_VERSION
> (0), DB version 100}}, then for each datanode "has apparent version ZDU (100)
> which is higher than SCM apparent version INITIAL_VERSION (0). SCM must
> finalize before datanodes." and "This datanode will shut down.", and finally
> all three datanodes stopped with the SCM caught up at applied index (t:2,
> i:74), still at INITIAL_VERSION. One snapshot install on that SCM in total. A
> passing test means the defect is present.
> h3. Patch
> [^MC-3-finalize-scm-from-db-on-startup.patch], against
> 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef. It also applies to master at
> 268654683b4.
> On startup, right after {{ScmVersionManager}} is built in
> {{StorageContainerManager.initializeSystemManagers}}, the SCM now calls
> {{RatisBasedVersionManager.finalizeToDBVersionIfRequired}}. It finalizes only
> up to the version recorded in the DB, never to the software version of the
> binary, and logs a warning when it does. It is a no-op unless the DB records
> a version the VERSION file does not allow; on a normal restart the DB is at
> or below the VERSION file, since the FINALIZE apply writes the VERSION file
> before the buffered {{#LAYOUTVERSION}}. It runs before the Ratis server and
> the datanode server start. The SCM upgrade actions it may run are safe at
> that point, and the {{ScmUpgradeAction}} javadoc now notes that an action may
> run during SCM construction, before the managers created after the version
> manager exist. An SCM whose {{#LAYOUTVERSION}} holds a version the binary
> does not know now fails at startup instead of ignoring the DB value, as with
> HDDS-16863 on OM.
> Related: HDDS-16863 (the same gap on OM) and HDDS-16862 (bound finalization
> from a snapshot). {{finalizeToDBVersionIfRequired}} and the bounded
> {{ComponentVersionManager.finalizeUpgrade(lastVersion)}} are the helpers
> proposed for OM in HDDS-16863, with identical hunks (HDDS-16862 adds the same
> {{finalizeUpgrade}} hunk). The patches merge cleanly in either order ({{git
> merge}}, or {{git apply --3way}} for the patch files); once HDDS-16863 is in,
> this patch reduces to the three lines in {{StorageContainerManager}}, the
> javadoc note and the test.
> Covered by {{testStartupFinalizesToApparentVersionInDB}} in the existing
> {{TestScmStartupInvalidApparentVersion}}. On one SCM storage it checks four
> cases:
> * Without a DB version the SCM stays pre-finalized.
> * A DB at {{ERASURE_CODED_STORAGE_SUPPORT}} over a VERSION file at
> {{INITIAL_VERSION}} finalizes the SCM to {{ERASURE_CODED_STORAGE_SUPPORT}}
> and no further.
> * A DB behind the VERSION file changes nothing.
> * A DB at the software version finalizes the SCM fully.
> Without the change it fails with {{expected: ERASURE_CODED_STORAGE_SUPPORT
> (3) but was: INITIAL_VERSION (0)}}. With it, these pass and checkstyle is
> clean:
> * {{TestScmStartupInvalidApparentVersion}} (3 tests)
> * the SCM, datanode and OM version manager suites,
> {{TestClearFinalizingStateScmUpgradeAction}}, {{TestSCMHAManagerImpl}},
> {{TestSCMStateMachine}}, the SCM construction and starter suites,
> {{TestDeletedBlockLog}} and {{TestBlockManager}} (145 tests)
> * {{TestScmHAFinalization}}, {{TestScmDataDistributionFinalization}} and
> {{TestSCMInstallSnapshotWithHA}} (8 tests)
> With the patch the reproduction fails, because the restarted SCM finalizes in
> its constructor ({{expected: INITIAL_VERSION (0) but was: ZDU (100)}}).
> Found by TLA+ model checking and code review of the SCM upgrade finalization
> paths under HDDS-15926, on commit 7f9b6237deb2b06d73d1ac2b78cccaa60780c3ef.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
