[
https://issues.apache.org/jira/browse/HDDS-17427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17427:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
2.1.0
1.4.1
2.1.1
1.3.0
2.2.0
> Datanode restarted during SchemaV3 finalization can create a second RocksDB
> for a data volume on another DB volume
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17427
> URL: https://issues.apache.org/jira/browse/HDDS-17427
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2,
> 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-5-reuse-existing-schema-v3-db-on-finalize-rerun.patch,
> TestBugMC5FinalizeRerunSecondDb.java
>
>
> h3. Mechanism
> {{ComponentVersionManager.finalizeUpgrade}} runs the upgrade action of a
> version and only then writes the new apparent version to the VERSION file. If
> the datanode stops between the two, it restarts below DATANODE_SCHEMA_V3,
> {{OzoneContainer}} loads no per volume RocksDB, and every data volume has
> {{dbParentDir == null}}. When SCM resends {{FinalizeVersionCommand}},
> {{DatanodeSchemaV3FinalizeAction}} calls {{HddsVolume.createDbStore}} again
> for every data volume. With {{hdds.datanode.container.db.dir}} set to two or
> more directories, {{createDbStore}} picks a DB volume at random and does not
> check whether one already holds {{<clusterID>/<storageID>/container.db}} for
> this data volume, so it can create a second RocksDB for the same storageID on
> another DB volume.
> Nothing reports the second copy. At every start
> {{HddsVolumeUtil.mapDbVolumesToDataVolumesIfNeeded}} maps the storageID to
> the last DB volume holding it in iteration order, and the finalize action
> itself reopens the DB through the same mapping, so while the set of DB
> volumes does not change all writes go to one copy and the other stays empty
> and unused. The javadoc of
> {{TestDatanodeUpgradeToSchemaV3.testFinalizeTwice}} ("finalize twice won't
> recreate any RocksDB") states the intent; that test calls {{finalizeUpgrade}}
> again only after the first call succeeded, so the action does not run a
> second time.
> h3. Trigger
> # A datanode with two or more DB volumes and SchemaV3 enabled is finalized
> for DATANODE_SCHEMA_V3. This happens only once, on a datanode first installed
> with a release before 1.3.0.
> # It stops after the action created the DBs and before the VERSION file is
> written (killed, or the VERSION write failed and the datanode was restarted
> before a later retry succeeded).
> # After the restart the action runs again and picks another DB volume for a
> data volume (probability 1/2 per re-run with two DB volumes).
> # Later the DB volume holding the copy in use is lost or removed from
> {{hdds.datanode.container.db.dir}}, and the datanode restarts.
> A failed VERSION write without a restart is not enough: the retry in the same
> process skips the volume.
> h3. Impact
> After step 4 the data volume does not fail. It maps to the other, empty copy
> and loads it without an error. Its SchemaV3 containers load with no blocks
> (they keep the state in their {{.container}} file, so a CLOSED container
> stays CLOSED), GetBlock for their blocks returns {{NO_SUCH_BLOCK}}, and the
> metadata and data scanners find nothing wrong. With a single copy the same
> loss leaves the containers unloaded, so the replica is visibly missing. That
> SCM then counts the empty CLOSED replica as a healthy replica and does not
> re-replicate the container is argued from {{RatisContainerReplicaCount}}, not
> run. Before step 4 the second copy only takes space.
> h3. Reproduction
> PASS on unmodified source. [^TestBugMC5FinalizeRerunSecondDb.java]
> ({{hadoop-hdds/container-service}}) runs a real {{DatanodeStateMachine}} with
> one data volume and two DB volumes, pre-finalized at
> ERASURE_CODED_STORAGE_SUPPORT. The stop between action and VERSION write is
> produced by making the layout version directory read only during
> {{finalizeUpgrade}} (the real action runs, the real VERSION write fails),
> followed by a normal restart, repeated until a second copy exists (at most 30
> tries, 4 in the recorded run). It then finalizes, writes and closes a
> container with one block, restarts once with the same configuration (block
> still there), and once after removing the DB volume in use from
> {{hdds.datanode.container.db.dir}}: the container is CLOSED with
> {{blockCount=0}}, GetBlock returns {{NO_SUCH_BLOCK}}, both scans report no
> errors. A passing test means the defect is present. A real disk failure of
> the DB volume takes the same path in the code but was not run.
> h3. Patch
> [^MC-5-reuse-existing-schema-v3-db-on-finalize-rerun.patch], against
> 679c9ccbee740a0af3ca9f76fc2d4bc6586049eb. It also applies to master at
> 69d77beff60.
> {{HddsVolume.createDbStore}} reuses the DB volume that already holds a
> directory for this storageID ({{DbVolume.getHddsVolumeIDs}}, filled by the DB
> volume's directory scan at startup) and picks one at random only when there
> is none. RocksDB then opens the existing instance. The placement of new DBs
> (first finalization, new volumes) is unchanged. Not covered: the reuse looks
> only at healthy DB volumes, so a DB volume holding the copy that is failed
> when the action runs again still gets a second copy elsewhere, and so does a
> data volume whose first DB was created on the data volume itself when DB
> volumes are added between the crashed run and the re-run. Both need a second
> event in the same window. Reordering the finalization to write VERSION first
> was not chosen, because a stop in between would then leave a datanode at
> DATANODE_SCHEMA_V3 without any DB.
> Covered by {{testCreateDbStoreAfterRestartReusesExistingDb}} in the existing
> {{TestHddsVolumeUtil}} (three data volumes, three DB volumes, two simulated
> restarts, each followed by {{createDbStore}}), which fails without the
> change. With it the volume, volume utility and upgrade test packages of
> {{hadoop-hdds/container-service}} plus {{TestOzoneContainer}} and
> {{TestContainerReader}} pass (303 tests), checkstyle is clean. With the patch
> the reproduction fails, because no second copy appears in 30 tries. Datanodes
> that already have a duplicate are not changed by the patch.
> Found by TLA+ model checking and code review of the SCM and datanode upgrade
> finalization paths under HDDS-15926, on commit
> 679c9ccbee740a0af3ca9f76fc2d4bc6586049eb. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
