Siyao Meng created HDDS-17427:
---------------------------------
Summary: Datanode restarted during SchemaV3 finalization can
create a second RocksDB for a data volume on another DB volume
Key: HDDS-17427
URL: https://issues.apache.org/jira/browse/HDDS-17427
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-reuse-existing-schema-v3-db-on-finalize-rerun.patch,
TestBugMC5FinalizeRerunSecondDb.java
h3. Mechanism
{{ComponentVersionManager.finalizeUpgrade}} runs the upgrade action of a
version and only then writes the new apparent version to the VERSION file. If
the datanode stops between the two, it restarts below DATANODE_SCHEMA_V3,
{{OzoneContainer}} loads no per volume RocksDB, and every data volume has
{{dbParentDir == null}}. When SCM resends {{FinalizeVersionCommand}},
{{DatanodeSchemaV3FinalizeAction}} calls {{HddsVolume.createDbStore}} again for
every data volume. With {{hdds.datanode.container.db.dir}} set to two or more
directories, {{createDbStore}} picks a DB volume at random and does not check
whether one already holds {{<clusterID>/<storageID>/container.db}} for this
data volume, so it can create a second RocksDB for the same storageID on
another DB volume.
Nothing reports the second copy. At every start
{{HddsVolumeUtil.mapDbVolumesToDataVolumesIfNeeded}} maps the storageID to the
last DB volume holding it in iteration order, and the finalize action itself
reopens the DB through the same mapping, so while the set of DB volumes does
not change all writes go to one copy and the other stays empty and unused. The
javadoc of {{TestDatanodeUpgradeToSchemaV3.testFinalizeTwice}} ("finalize twice
won't recreate any RocksDB") states the intent; that test calls
{{finalizeUpgrade}} again only after the first call succeeded, so the action
does not run a second time.
h3. Trigger
# A datanode with two or more DB volumes and SchemaV3 enabled is finalized for
DATANODE_SCHEMA_V3. This happens only once, on a datanode first installed with
a release before 1.3.0.
# It stops after the action created the DBs and before the VERSION file is
written (killed, or the VERSION write failed and the datanode was restarted
before a later retry succeeded).
# After the restart the action runs again and picks another DB volume for a
data volume (probability 1/2 per re-run with two DB volumes).
# Later the DB volume holding the copy in use is lost or removed from
{{hdds.datanode.container.db.dir}}, and the datanode restarts.
A failed VERSION write without a restart is not enough: the retry in the same
process skips the volume.
h3. Impact
After step 4 the data volume does not fail. It maps to the other, empty copy
and loads it without an error. Its SchemaV3 containers load with no blocks
(they keep the state in their {{.container}} file, so a CLOSED container stays
CLOSED), GetBlock for their blocks returns {{NO_SUCH_BLOCK}}, and the metadata
and data scanners find nothing wrong. With a single copy the same loss leaves
the containers unloaded, so the replica is visibly missing. That SCM then
counts the empty CLOSED replica as a healthy replica and does not re-replicate
the container is argued from {{RatisContainerReplicaCount}}, not run. Before
step 4 the second copy only takes space.
h3. Reproduction
PASS on unmodified source. [^TestBugMC5FinalizeRerunSecondDb.java]
({{hadoop-hdds/container-service}}) runs a real {{DatanodeStateMachine}} with
one data volume and two DB volumes, pre-finalized at
ERASURE_CODED_STORAGE_SUPPORT. The stop between action and VERSION write is
produced by making the layout version directory read only during
{{finalizeUpgrade}} (the real action runs, the real VERSION write fails),
followed by a normal restart, repeated until a second copy exists (at most 30
tries, 4 in the recorded run). It then finalizes, writes and closes a container
with one block, restarts once with the same configuration (block still there),
and once after removing the DB volume in use from
{{hdds.datanode.container.db.dir}}: the container is CLOSED with
{{blockCount=0}}, GetBlock returns {{NO_SUCH_BLOCK}}, both scans report no
errors. A passing test means the defect is present. A real disk failure of the
DB volume takes the same path in the code but was not run.
h3. Patch
[^MC-5-reuse-existing-schema-v3-db-on-finalize-rerun.patch], against
679c9ccbee740a0af3ca9f76fc2d4bc6586049eb. It also applies to master at
69d77beff60.
{{HddsVolume.createDbStore}} reuses the DB volume that already holds a
directory for this storageID ({{DbVolume.getHddsVolumeIDs}}, filled by the DB
volume's directory scan at startup) and picks one at random only when there is
none. RocksDB then opens the existing instance. The placement of new DBs (first
finalization, new volumes) is unchanged. Not covered: the reuse looks only at
healthy DB volumes, so a DB volume holding the copy that is failed when the
action runs again still gets a second copy elsewhere, and so does a data volume
whose first DB was created on the data volume itself when DB volumes are added
between the crashed run and the re-run. Both need a second event in the same
window. Reordering the finalization to write VERSION first was not chosen,
because a stop in between would then leave a datanode at DATANODE_SCHEMA_V3
without any DB.
Covered by {{testCreateDbStoreAfterRestartReusesExistingDb}} in the existing
{{TestHddsVolumeUtil}} (three data volumes, three DB volumes, two simulated
restarts, each followed by {{createDbStore}}), which fails without the change.
With it the volume, volume utility and upgrade test packages of
{{hadoop-hdds/container-service}} plus {{TestOzoneContainer}} and
{{TestContainerReader}} pass (303 tests), checkstyle is clean. With the patch
the reproduction fails, because no second copy appears in 30 tries. Datanodes
that already have a duplicate are not changed by the patch.
Found by TLA+ model checking and code review of the SCM and datanode upgrade
finalization paths under HDDS-15926, on commit
679c9ccbee740a0af3ca9f76fc2d4bc6586049eb. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
