Siyao Meng created HDDS-16860:
---------------------------------
Summary: OM leader terminates on CompleteFinalizeUpgrade when a
quota repair is already running, and stays partially finalized after restart
Key: HDDS-16860
URL: https://issues.apache.org/jira/browse/HDDS-16860
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-skip-upgrade-quota-repair-when-already-running.patch,
TestBugMC2QuotaRepairStuckFinalizing.java
h3. Mechanism
{{QuotaRepairUpgradeAction.execute}} runs on the OM that is the ready leader
when it applies {{CompleteFinalizeUpgrade}} and calls
{{QuotaRepairTask.repair}}. If a quota repair is already running on that OM,
{{repair}} throws {{QUOTA_ERROR}} "Quota repair is already running". The action
only catches {{OMNotLeaderException}} and {{OMLeaderNotReadyException}}, so
{{OMVersionManager.runUpgradeAction}} wraps the error in an
{{UpgradeException}} ({{FINALIZE_UPGRADE_ACTION_FAILED}}). That is not an
{{OMException}}, so the response status is {{INTERNAL_ERROR}} and
{{OzoneManagerStateMachine.processResponse}} terminates the OM.
At that point the versions before QUOTA have already been written to the
VERSION file. The error response is also already in the double buffer, which
skips its DB batch but still records its transaction index. The restarted OM
therefore resumes after the failed entry and never applies it again. It stays
at {{FILESYSTEM_SNAPSHOT}} (5) with the finalization in progress key set, while
the followers, which skip the action, finalize to the software version.
h3. Trigger
# A cluster is pre-finalized at a layout before QUOTA, with
{{ozone.om.upgrade.quota.recalculate.enabled}} at its default ({{true}}).
# An admin starts a quota repair ({{ozone repair om quota start}}, which runs
on the leader).
# While it is still running, the admin finalizes the upgrade and the leader
applies {{CompleteFinalizeUpgrade}}.
No fault is needed. On a large cluster one repair can run for a long time; the
reproduction keeps one running by calling {{startQuotaRepair}} in a loop.
h3. Impact
* The leader OM terminates during the upgrade (run). The other OMs elect a new
leader and keep serving.
* After a restart over the same storage, that OM has applied past
{{CompleteFinalizeUpgrade}} but reports {{FILESYSTEM_SNAPSHOT}} in memory and
in its VERSION file, with the finalization in progress key still in its DB,
while its peers are at the software version (run).
* Argued, not run: a new OM process recreates {{OMUpgradeFinalizeService}}
because it still needs finalization. That service submits
{{CompleteFinalizeUpgrade}} only while the OM is the ready leader, so the OM
stays behind until leadership moves to it or it installs a Ratis snapshot from
the leader ({{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}).
Restarting it does not help. While behind, it reports an older version to
clients and keeps version gated features such as snapshot defrag off.
* In the reproduction the double buffer always flushed the failed entry before
the OM was stopped. With a real process exit this is a race. If the exit wins,
the entry is applied again on restart and is expected to finalize.
* Affects 2.0.0 onward for the termination.
{{QuotaRepairUpgradeAction.execute}} is the same since 2.0.0, and in 2.x it
runs while the leader applies {{FinalizeUpgrade}}, so the same error is
expected to fail that request with {{INTERNAL_ERROR}} and terminate the leader
(argued from code, not run). The partially finalized state after restart is
specific to master, because {{CompleteFinalizeUpgrade}} and
{{OMUpgradeFinalizeService}} are not in any release.
h3. Reproduction
PASS on the unmodified source. [^TestBugMC2QuotaRepairStuckFinalizing.java]
({{hadoop-ozone/integration-test}}) runs a 3 OM HA mini cluster pre-finalized
at {{INITIAL_VERSION}} and uses only admin RPCs ({{startQuotaRepair}} in a
loop, then {{finalizeUpgrade}}). The mini cluster disables {{System.exit}}, so
the test stops the terminated OM at once, as a process exit would, and restarts
it with {{restartOzoneManager}}. The test asserts:
* the exit message names {{CompleteFinalizeUpgrade}} and "Quota repair is
already running";
* the restarted OM has applied past that entry but is still at
{{FILESYSTEM_SNAPSHOT}} with the marker set;
* the peers are at the software version.
A passing test means the defect is present. Recovery after a leadership change
is not covered, because {{OzoneManager.restart}} (used by the mini cluster)
does not restart {{OMUpgradeFinalizeService}}.
h3. Patch
[^MC-2-skip-upgrade-quota-repair-when-already-running.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{QuotaRepairUpgradeAction}} now treats {{QUOTA_ERROR}} from
{{QuotaRepairTask.repair}} like the existing not leader case. It logs a WARN
and lets finalization continue, and rethrows any other {{OMException}}. The
action only starts an asynchronous repair whose result finalization never waits
for, so a repair that is already running is a reason to skip, not to fail. The
WARN asks the operator to run quota repair again for all buckets if the running
one was limited with {{--buckets}}.
Covered by {{testUpgradeActionDoesNotFailWhenQuotaRepairIsRunning}} in the
existing {{TestQuotaRepairTask}}. It holds a real repair inside its Ratis
submit and runs the action as the ready leader. Without the change it fails
with the {{QUOTA_ERROR}} thrown from {{QuotaRepairUpgradeAction.execute}}. With
it, these pass:
* {{TestQuotaRepairTask}}, {{TestOMVersionManager}},
{{TestOMUpgradeFinalizeService}} and {{TestOMCompleteFinalizeUpgradeRequest}}
(48 tests);
* {{TestOMUpgradeFinalization}}, {{TestOMBucketLayoutUpgrade}} and
{{TestUpgradeFinalizationWithOldClients}} on a mini cluster (12 tests).
Checkstyle is clean. With the patch the reproduction fails at the wait for the
OM to terminate: the leader logs the skip and all OMs finalize.
The patch removes this trigger only. Any write whose apply returns
{{INTERNAL_ERROR}} can have its index flushed before the process exits. It is
then not applied again after restart. For {{CompleteFinalizeUpgrade}} that now
needs an I/O failure in an upgrade action or in the VERSION file write.
Two related fixes in {{QuotaRepairUpgradeAction}}, filed separately and linked
from there, change the same few lines of {{QuotaRepairUpgradeAction.execute}},
so whichever lands later needs a trivial rebase.
Found by TLA+ model checking and code review of the OM upgrade finalization
paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
