[
https://issues.apache.org/jira/browse/HDDS-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17422:
------------------------------
Attachment: MC-2-bootstrapping-om-adopts-peer-apparent-version.patch
> OM bootstrapped into a pre-finalized OM ring runs at its software version,
> exposes unfinalized features and stalls finalization while it leads
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17422
> URL: https://issues.apache.org/jira/browse/HDDS-17422
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.3.0,
> 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-bootstrapping-om-adopts-peer-apparent-version.patch,
> TestBugMC2FreshBootstrappedOmPreFinalizedRing.java
>
>
> h3. Mechanism
> A new OM initializes its storage with a VERSION file at
> {{OzoneManagerVersion.SOFTWARE_VERSION}} ({{OMStorage}} constructor). When it
> joins an existing ring with {{ozone om --bootstrap}}, nothing moves it to the
> ring's apparent version: the bootstrap handshake does not carry a version,
> {{RatisBasedVersionManager.validateDBVersion}} only logs a mismatch, and
> {{finalizeFromSnapshotIfRequired}} can only raise the version. A ring that
> has not been finalized since it was installed has no {{#LAYOUTVERSION}} in
> its DB at all. So if the ring is pre-finalized, the new OM is finalized on
> its own while its peers are not. The ZDU design requires all OMs of a ring to
> run at the same apparent version.
> The rest follows from that one replica:
> * {{OzoneManager.getServiceList}} reports the local apparent version for
> every OM, on the assumption that peers share it.
> * {{OMUpgradeFinalizeService}} is only created when the local OM needs
> finalization, and only the leader's instance submits
> {{CompleteFinalizeUpgrade}}. The new OM has none.
> * {{OzoneManager.queryUpgradeStatus}} reports the answering OM's own state.
> h3. Trigger
> # The OM ring runs the new software and is not finalized yet (apparent
> version ZDU, or a legacy layout version after an upgrade from an older
> release).
> # An operator adds or replaces an OM with {{ozone om --bootstrap}} (which
> initializes the OM storage first), or with {{ozone om --bootstrap --force}}
> while a peer is down.
> # The new OM becomes leader through an election or {{ozone admin om
> transfer}}.
> The documented recovery of a faulty OM reaches the same state: delete its DB
> and VERSION file, run {{ozone om --init}}, and restart it normally
> (HDDS-16534). That path does not use bootstrap (see Patch).
> h3. Impact
> * While the new OM leads, a client that connects learns the new OM's version
> from the service list and keeps it for its lifetime. {{RpcClient}} then
> allows If-None-Match writes, and they are accepted although three of four OMs
> are not finalized for them (run). An S3 Gateway that first connects in that
> window would do the same (argued, not run).
> * If the admin finalizes while the new OM leads, SCM finalizes and the other
> OMs record the in-progress marker, but no {{CompleteFinalizeUpgrade}} is ever
> submitted. The other OMs stay pre-finalized, and {{queryUpgradeStatus}}
> reports FINALIZED for the cluster and for OM (run, 5 s window with a 10 ms
> check interval). Moving leadership to an original OM completes finalization.
> * Features gated on the new version are usable before finalization, so the
> downgrade that the pre-finalized state promises is no longer safe (argued).
> Apart from the finalization in progress marker, which only the original OMs
> write, I found no replica divergence: with identical software, each replica
> evaluates If-None-Match at apply time. A replica on older software would
> ignore the unknown field (argued, not run).
> * Releases 1.3.0 through 2.2.1 have the same root cause in an older form
> (read at the release tags, not run). A fresh OM starts at the newest layout
> version and bootstrap does not lower it, so while it leads a pre-finalized
> ring, it admits requests gated by {{@DisallowedUntilLayoutVersion}} before
> finalization. There finalization is applied on every replica, so it does not
> stall, and the service list reports the software version.
> h3. Reproduction
> PASS on unmodified source.
> [^TestBugMC2FreshBootstrappedOmPreFinalizedRing.java]
> ({{hadoop-ozone/integration-test}}) starts a mini cluster with three OMs at
> apparent version ZDU (test only
> {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}}) and one datanode. It removes
> the key, bootstraps a fourth OM through the normal init and bootstrap path,
> and transfers leadership to it. Output:
> {noformat}
> Bootstrapped omNode-bootstrap-1 at apparent version S3_IF_NONE_MATCH_ETAG
> (104); peers at [ZDU (100), ZDU (100), ZDU (100)]; its service list reports
> [S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104),
> S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104)]
> If-None-Match write accepted while 3 of 4 OMs are at apparent version ZDU
> (100)
> After 5000 ms: leader omNode-bootstrap-1, cluster status FINALIZED, OM status
> FINALIZED, peers at [ZDU (100), ZDU (100), ZDU (100)]
> After leadership moved to omNode-3, the original OMs finalized to
> [S3_IF_NONE_MATCH_ETAG (104), S3_IF_NONE_MATCH_ETAG (104),
> S3_IF_NONE_MATCH_ETAG (104)]
> {noformat}
> As a control, the If-None-Match write is refused client side before the
> bootstrap. A passing test means the defect is present.
> h3. Patch
> [^MC-2-bootstrapping-om-adopts-peer-apparent-version.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
> e03ba37c41d.
> When an OM starts with {{ozone om --bootstrap}} (with or without {{--force}})
> and has no {{om.db}} yet, it asks each configured peer for its apparent
> version before it builds its version manager, and lowers its VERSION file to
> the lowest answer. Lowering is safe because nothing has been persisted yet. A
> lagging peer can only make it start lower, and finalization over Ratis or
> from a snapshot then raises it like any follower (how far that raise goes is
> HDDS-16862, which also applies to an older {{CompleteFinalizeUpgrade}} entry
> replayed from a Raft log that was never purged). The peers answer through the
> OM admin RPC {{getPeerUpgradeStatus}}, which gets a new optional field
> {{omApparentVersion}}. That RPC is OM to OM, is not applied through Ratis,
> and was added after 2.2.0.
> The query does not retry connections or socket timeouts and uses a 5 s RPC
> timeout. The client's fixed retry of 10 attempts at 1 s still applies. A
> stopped peer costs about 10 s (measured in a force bootstrap with one OM
> down). A peer host that does not answer at all can cost up to 11 times
> {{ipc.client.connect.timeout}} (20 s by default), about 4 minutes
> (calculated, not run). With default settings and no bound, it would be
> minutes per stopped peer and up to hours per silent host (calculated). If no
> peer reports a version, the OM logs a warning and keeps today's behaviour.
> Refusing to bootstrap in that case would be the stricter alternative.
> Not covered: the re-init and normal restart path in Trigger. I ran it on the
> patched code with a ring at a legacy layout version. The re-initialized OM
> caught up at {{S3_IF_NONE_MATCH_ETAG (104)}}, and after a leadership transfer
> it led at 104 and reported 104 for every OM. A follow-up would run the same
> bounded query on a regular HA start without an {{om.db}}, falling back to the
> software version when no peer answers, because the first start of a new ring
> has no peers up yet. That changes the first start of every new HA cluster, so
> it is left out of this patch.
> Covered by {{testBootstrappedOmAdoptsApparentVersionOfPreFinalizedRing}} in
> the existing {{TestOMUpgradeFinalization}}. It bootstraps an OM into a ring
> at a legacy layout version, makes it leader and finalizes. Without the change
> it fails with {{expected: INITIAL_VERSION (0) but was: S3_IF_NONE_MATCH_ETAG
> (104)}}. {{TestOMAdminProtocolServerSideImpl}} checks the new field. With the
> patch, these pass, and so do {{TestOMUpgradeFinalization}} (4 tests),
> {{TestAddRemoveOzoneManager}} (8 tests, including force bootstrap with an OM
> down) and the OM upgrade and startup unit suites (81 tests); checkstyle is
> clean. The reproduction then fails at its first assertion, because the new OM
> starts at ZDU.
> Found by TLA+ model checking and code review of the zero downtime upgrade
> version gating in SCM, OM and the datanode write path under HDDS-15926, on
> commit 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing; HDDS-16862 and
> HDDS-17140 are related but different. The attached patch is a proposal for
> review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
