Siyao Meng created HDDS-17459:
---------------------------------
Summary: OM restart re-creates a deleted S3 volume outside Ratis
and replaces the OM user's volume list, so OMs diverge
Key: HDDS-17459
URL: https://issues.apache.org/jira/browse/HDDS-17459
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-do-not-recreate-s3-volume-outside-ratis-on-restart.patch,
TestBugMC3DeletedS3VolumeRecreatedOnRestart.java
h3. Mechanism
The {{OzoneManager}} constructor calls {{addS3GVolumeToDB}} before the Ratis
server is set up. If the volume table has no entry for the S3 volume
({{ozone.s3g.volume.name}}, default {{s3v}}), it commits the volume and a new
{{userTable}} entry for the OM service user, holding only the S3 volume,
directly to RocksDB and the table caches. No log entry is involved, and the
user entry replaces the user's existing volume list instead of adding to it.
{{OMVolumeDeleteRequest}} has no special case for the S3 volume, so an empty
{{s3v}} can be deleted through Ratis like any other volume. After that, every
OM process restart runs the constructor again and writes the deleted volume and
the one entry user list on that OM only.
h3. Trigger
# The empty {{s3v}} is deleted through the client ({{ozone sh volume delete
/s3v}}), for example on a cluster that does not use S3.
# Any OM process is restarted.
The volume list part needs the OM service user to own another volume, for
example one created on a cluster where clients run as that user.
No fault is needed. The mini cluster's {{restartOzoneManager}} reuses the
{{OzoneManager}} instance and skips the constructor, so the existing restart
tests do not show it.
h3. Impact
Run on a three OM mini cluster on unmodified source, restarting one follower
with {{OzoneManager.createOm(conf)}} plus {{start()}}:
* After the restart the restarted OM has {{s3v}} and
{{userTable[omUser]=[s3v]}}, while its peers have no {{s3v}} and {{[vol1]}}.
All three are at the same applied index.
* With the restarted OM as leader, the client sees the deleted volume again,
and {{listVolumesByUser}} returns {{[s3v]}} without {{vol1}}.
* {{createBucket}} in {{s3v}} through that leader succeeds, and the bucket
exists on that OM only. Keys written to such a bucket would exist on one
replica (argued, not run).
* A replicated {{CreateVolume(s3v)}} is rejected with {{VOLUME_ALREADY_EXISTS}}
on the restarted OM and applied on the others, so it does not repair the
divergence.
Nothing reconciles state written outside the log; only a checkpoint install on
that OM replaces it. A single OM deployment is affected too: the OM user's
other volumes drop out of its list and {{s3v}} comes back with no log entry.
h3. Reproduction
[^TestBugMC3DeletedS3VolumeRecreatedOnRestart.java]
({{hadoop-ozone/integration-test}}, target path and Maven command in the class
comment) runs three OMs without datanodes, with volume and bucket operations
through the client API and leadership moved through the OM admin client: create
{{vol1}}, delete {{s3v}}, restart one follower by building a new
{{OzoneManager}} on its configuration, move leadership to it, then create a
bucket in {{s3v}} and create {{s3v}} again. It prints each OM's state after
waiting for all OMs to reach the same applied index and flushing their double
buffers. A passing test means the defect is present. With the patch it fails at
the check right after the restart, because the restarted OM keeps {{s3v=false
userTable=[vol1]}} like its peers.
h3. Suggested fix
Create the S3 volume at startup only on a new DB. An existing DB without it
lost it through the log (or the configured name was changed), and recreating it
belongs in the log as well ({{ozone sh volume create}}). Moving the creation
itself into a Ratis request submitted by the leader would also remove the
remaining per OM differences of the startup volume (creation time, group ACLs),
but needs new request handling.
h3. Patch
[^MC-3-do-not-recreate-s3-volume-outside-ratis-on-restart.patch], against
a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
b6d0181075e.
{{addS3GVolumeToDB}} now returns early, with one WARN line, when the volume is
missing and the DB already has a {{TransactionInfo}} row, that is, when
transactions have been applied to it. A new DB still gets the volume on every
OM as before. Because the user table of a new DB is empty, the list overwrite
cannot happen any more either. Operator visible change: after changing
{{ozone.s3g.volume.name}} on an existing cluster, the new volume has to be
created through the client instead of being created by each OM at restart.
Covered by {{testDeletedS3VolumeNotRecreatedByNewOMInstance}} in the existing
{{TestOzoneManagerRestart}}: it deletes {{s3v}}, starts a new {{OzoneManager}}
instance on the same DB, and checks through the client that {{s3v}} stays
deleted, that the OM user's other volume is still listed, and that {{s3v}} can
be created again. It fails without the change because {{s3v}} is back. With the
change {{TestOzoneManagerRestart}}, {{TestOmMetrics}},
{{TestAddRemoveOzoneManager}}, {{TestOzoneManagerHAWithStoppedNodes}} and the
{{ozone-manager}} suites that start an OM on a new DB ({{TestKeyManagerUnit}},
{{TestBucketManagerImpl}}, {{TestVolumeOwner}}, {{TestOzoneManagerStarter}})
pass (76 tests), checkstyle is clean. An OM that already diverged is not
repaired.
Found by TLA+ model checking and code review of the OM table cache, double
buffer flush and startup paths under HDDS-15926, on commit
a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing (nearest: HDDS-3993,
which added the startup creation, and HDDS-13823, about the cache entry of the
same write). The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
