Siyao Meng created HDDS-17422:
---------------------------------
Summary: OM bootstrapped into a pre-finalized OM ring runs at its
software version, exposes unfinalized features and stalls finalization while it
leads
Key: HDDS-17422
URL: https://issues.apache.org/jira/browse/HDDS-17422
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-2-bootstrapping-om-adopts-peer-apparent-version.patch,
TestBugMC2FreshBootstrappedOmPreFinalizedRing.java
h3. Mechanism
A new OM initializes its storage with a VERSION file at
{{OzoneManagerVersion.SOFTWARE_VERSION}} ({{OMStorage}} constructor). When it
joins an existing ring with {{ozone om --bootstrap}}, nothing moves it to the
ring's apparent version: the bootstrap handshake does not carry a version,
{{RatisBasedVersionManager.validateDBVersion}} only logs a mismatch, and
{{finalizeFromSnapshotIfRequired}} can only raise the version. A ring that has
not been finalized since it was installed has no {{#LAYOUTVERSION}} in its DB
at all. So if the ring is pre-finalized, the new OM is finalized on its own
while its peers are not. The ZDU design requires all OMs of a ring to run at
the same apparent version.
The rest follows from that one replica:
* {{OzoneManager.getServiceList}} reports the local apparent version for every
OM, on the assumption that peers share it.
* {{OMUpgradeFinalizeService}} is only created when the local OM needs
finalization, and only the leader's instance submits
{{CompleteFinalizeUpgrade}}. The new OM has none.
* {{OzoneManager.queryUpgradeStatus}} reports the answering OM's own state.
h3. Trigger
# The OM ring runs the new software and is not finalized yet (apparent version
ZDU, or a legacy layout version after an upgrade from an older release).
# An operator adds or replaces an OM with {{ozone om --bootstrap}} (which
initializes the OM storage first), or with {{ozone om --bootstrap --force}}
while a peer is down.
# The new OM becomes leader through an election or {{ozone admin om transfer}}.
The documented recovery of a faulty OM reaches the same state: delete its DB
and VERSION file, run {{ozone om --init}}, and restart it normally
(HDDS-16534). That path does not use bootstrap (see Patch).
h3. Impact
* While the new OM leads, a client that connects learns the new OM's version
from the service list and keeps it for its lifetime. {{RpcClient}} then allows
If-None-Match writes, and they are accepted although three of four OMs are not
finalized for them (run). An S3 Gateway that first connects in that window
would do the same (argued, not run).
* If the admin finalizes while the new OM leads, SCM finalizes and the other
OMs record the in-progress marker, but no {{CompleteFinalizeUpgrade}} is ever
submitted. The other OMs stay pre-finalized, and {{queryUpgradeStatus}} reports
FINALIZED for the cluster and for OM (run, 5 s window with a 10 ms check
interval). Moving leadership to an original OM completes finalization.
* Features gated on the new version are usable before finalization, so the
downgrade that the pre-finalized state promises is no longer safe (argued).
Apart from the finalization in progress marker, which only the original OMs
write, I found no replica divergence: with identical software, each replica
evaluates If-None-Match at apply time. A replica on older software would ignore
the unknown field (argued, not run).
* Releases 1.3.0 through 2.2.1 have the same root cause in an older form (read
at the release tags, not run). A fresh OM starts at the newest layout version
and bootstrap does not lower it, so while it leads a pre-finalized ring, it
admits requests gated by {{@DisallowedUntilLayoutVersion}} before finalization.
There finalization is applied on every replica, so it does not stall, and the
service list reports the software version.
h3. Reproduction
PASS on unmodified source.
[^TestBugMC2FreshBootstrappedOmPreFinalizedRing.java]
({{hadoop-ozone/integration-test}}) starts a mini cluster with three OMs at
apparent version ZDU (test only
{{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}) and one datanode. It removes
the key, bootstraps a fourth OM through the normal init and bootstrap path, and
transfers leadership to it. Output:
{noformat}
Bootstrapped omNode-bootstrap-1 at apparent version S3_IF_NONE_MATCH_ETAG
(104); peers at [ZDU (100), ZDU (100), ZDU (100)]; its service list reports
[S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104),
S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104)]
If-None-Match write accepted while 3 of 4 OMs are at apparent version ZDU (100)
After 5000 ms: leader omNode-bootstrap-1, cluster status FINALIZED, OM status
FINALIZED, peers at [ZDU (100), ZDU (100), ZDU (100)]
After leadership moved to omNode-3, the original OMs finalized to
[S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104),
S3_IF_NONE_MATCH_ETAG (104)]
{noformat}
As a control, the If-None-Match write is refused client side before the
bootstrap. A passing test means the defect is present.
h3. Patch
[^MC-2-bootstrapping-om-adopts-peer-apparent-version.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
e03ba37c41d.
When an OM starts with {{ozone om --bootstrap}} (with or without {{--force}})
and has no {{om.db}} yet, it asks each configured peer for its apparent version
before it builds its version manager, and lowers its VERSION file to the lowest
answer. Lowering is safe because nothing has been persisted yet. A lagging peer
can only make it start lower, and finalization over Ratis or from a snapshot
then raises it like any follower (how far that raise goes is HDDS-16862, which
also applies to an older {{CompleteFinalizeUpgrade}} entry replayed from a Raft
log that was never purged). The peers answer through the OM admin RPC
{{getPeerUpgradeStatus}}, which gets a new optional field
{{omApparentVersion}}. That RPC is OM to OM, is not applied through Ratis, and
was added after 2.2.0.
The query does not retry connections or socket timeouts and uses a 5 s RPC
timeout. The client's fixed retry of 10 attempts at 1 s still applies. A
stopped peer costs about 10 s (measured in a force bootstrap with one OM down).
A peer host that does not answer at all can cost up to 11 times
{{ipc.client.connect.timeout}} (20 s by default), about 4 minutes (calculated,
not run). With default settings and no bound, it would be minutes per stopped
peer and up to hours per silent host (calculated). If no peer reports a
version, the OM logs a warning and keeps today's behaviour. Refusing to
bootstrap in that case would be the stricter alternative.
Not covered: the re-init and normal restart path in Trigger. I ran it on the
patched code with a ring at a legacy layout version. The re-initialized OM
caught up at {{S3_IF_NONE_MATCH_ETAG (104)}}, and after a leadership transfer
it led at 104 and reported 104 for every OM. A follow-up would run the same
bounded query on a regular HA start without an {{om.db}}, falling back to the
software version when no peer answers, because the first start of a new ring
has no peers up yet. That changes the first start of every new HA cluster, so
it is left out of this patch.
Covered by {{testBootstrappedOmAdoptsApparentVersionOfPreFinalizedRing}} in the
existing {{TestOMUpgradeFinalization}}. It bootstraps an OM into a ring at a
legacy layout version, makes it leader and finalizes. Without the change it
fails with {{expected: INITIAL_VERSION (0) but was: S3_IF_NONE_MATCH_ETAG
(104)}}. {{TestOMAdminProtocolServerSideImpl}} checks the new field. With the
patch, these pass, and so do {{TestOMUpgradeFinalization}} (4 tests),
{{TestAddRemoveOzoneManager}} (8 tests, including force bootstrap with an OM
down) and the OM upgrade and startup unit suites (81 tests); checkstyle is
clean. The reproduction then fails at its first assertion, because the new OM
starts at ZDU.
Found by TLA+ model checking and code review of the zero downtime upgrade
version gating in SCM, OM and the datanode write path under HDDS-15926, on
commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues
and apache/ozone pull requests for duplicates before filing; HDDS-16862 and
HDDS-17140 are related but different. The attached patch is a proposal for
review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
