[
https://issues.apache.org/jira/browse/HDDS-16860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16860:
------------------------------
Attachment: TestBugMC2QuotaRepairStuckFinalizing.java
> OM leader terminates on CompleteFinalizeUpgrade when a quota repair is
> already running, and stays partially finalized after restart
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16860
> URL: https://issues.apache.org/jira/browse/HDDS-16860
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-skip-upgrade-quota-repair-when-already-running.patch,
> TestBugMC2QuotaRepairStuckFinalizing.java
>
>
> h3. Mechanism
> {{QuotaRepairUpgradeAction.execute}} runs on the OM that is the ready leader
> when it applies {{CompleteFinalizeUpgrade}} and calls
> {{QuotaRepairTask.repair}}. If a quota repair is already running on that OM,
> {{repair}} throws {{QUOTA_ERROR}} "Quota repair is already running". The
> action only catches {{OMNotLeaderException}} and
> {{OMLeaderNotReadyException}}, so {{OMVersionManager.runUpgradeAction}} wraps
> the error in an {{UpgradeException}} ({{FINALIZE_UPGRADE_ACTION_FAILED}}).
> That is not an {{OMException}}, so the response status is {{INTERNAL_ERROR}}
> and {{OzoneManagerStateMachine.processResponse}} terminates the OM.
> At that point the versions before QUOTA have already been written to the
> VERSION file. The error response is also already in the double buffer, which
> skips its DB batch but still records its transaction index. The restarted OM
> therefore resumes after the failed entry and never applies it again. It stays
> at {{FILESYSTEM_SNAPSHOT}} (5) with the finalization in progress key set,
> while the followers, which skip the action, finalize to the software version.
> h3. Trigger
> # A cluster is pre-finalized at a layout before QUOTA, with
> {{ozone.om.upgrade.quota.recalculate.enabled}} at its default ({{true}}).
> # An admin starts a quota repair ({{ozone repair om quota start}}, which runs
> on the leader).
> # While it is still running, the admin finalizes the upgrade and the leader
> applies {{CompleteFinalizeUpgrade}}.
> No fault is needed. On a large cluster one repair can run for a long time;
> the reproduction keeps one running by calling {{startQuotaRepair}} in a loop.
> h3. Impact
> * The leader OM terminates during the upgrade (run). The other OMs elect a
> new leader and keep serving.
> * After a restart over the same storage, that OM has applied past
> {{CompleteFinalizeUpgrade}} but reports {{FILESYSTEM_SNAPSHOT}} in memory and
> in its VERSION file, with the finalization in progress key still in its DB,
> while its peers are at the software version (run).
> * Argued, not run: a new OM process recreates {{OMUpgradeFinalizeService}}
> because it still needs finalization. That service submits
> {{CompleteFinalizeUpgrade}} only while the OM is the ready leader, so the OM
> stays behind until leadership moves to it or it installs a Ratis snapshot
> from the leader
> ({{RatisBasedVersionManager.finalizeFromSnapshotIfRequired}}). Restarting it
> does not help. While behind, it reports an older version to clients and keeps
> version gated features such as snapshot defrag off.
> * In the reproduction the double buffer always flushed the failed entry
> before the OM was stopped. With a real process exit this is a race. If the
> exit wins, the entry is applied again on restart and is expected to finalize.
> * Affects 2.0.0 onward for the termination.
> {{QuotaRepairUpgradeAction.execute}} is the same since 2.0.0, and in 2.x it
> runs while the leader applies {{FinalizeUpgrade}}, so the same error is
> expected to fail that request with {{INTERNAL_ERROR}} and terminate the
> leader (argued from code, not run). The partially finalized state after
> restart is specific to master, because {{CompleteFinalizeUpgrade}} and
> {{OMUpgradeFinalizeService}} are not in any release.
> h3. Reproduction
> PASS on the unmodified source. [^TestBugMC2QuotaRepairStuckFinalizing.java]
> ({{hadoop-ozone/integration-test}}) runs a 3 OM HA mini cluster pre-finalized
> at {{INITIAL_VERSION}} and uses only admin RPCs ({{startQuotaRepair}} in a
> loop, then {{finalizeUpgrade}}). The mini cluster disables {{System.exit}},
> so the test stops the terminated OM at once, as a process exit would, and
> restarts it with {{restartOzoneManager}}. The test asserts:
> * the exit message names {{CompleteFinalizeUpgrade}} and "Quota repair is
> already running";
> * the restarted OM has applied past that entry but is still at
> {{FILESYSTEM_SNAPSHOT}} with the marker set;
> * the peers are at the software version.
> A passing test means the defect is present. Recovery after a leadership
> change is not covered, because {{OzoneManager.restart}} (used by the mini
> cluster) does not restart {{OMUpgradeFinalizeService}}.
> h3. Patch
> [^MC-2-skip-upgrade-quota-repair-when-already-running.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{QuotaRepairUpgradeAction}} now treats {{QUOTA_ERROR}} from
> {{QuotaRepairTask.repair}} like the existing not leader case. It logs a WARN
> and lets finalization continue, and rethrows any other {{OMException}}. The
> action only starts an asynchronous repair whose result finalization never
> waits for, so a repair that is already running is a reason to skip, not to
> fail. The WARN asks the operator to run quota repair again for all buckets if
> the running one was limited with {{--buckets}}.
> Covered by {{testUpgradeActionDoesNotFailWhenQuotaRepairIsRunning}} in the
> existing {{TestQuotaRepairTask}}. It holds a real repair inside its Ratis
> submit and runs the action as the ready leader. Without the change it fails
> with the {{QUOTA_ERROR}} thrown from {{QuotaRepairUpgradeAction.execute}}.
> With it, these pass:
> * {{TestQuotaRepairTask}}, {{TestOMVersionManager}},
> {{TestOMUpgradeFinalizeService}} and {{TestOMCompleteFinalizeUpgradeRequest}}
> (48 tests);
> * {{TestOMUpgradeFinalization}}, {{TestOMBucketLayoutUpgrade}} and
> {{TestUpgradeFinalizationWithOldClients}} on a mini cluster (12 tests).
> Checkstyle is clean. With the patch the reproduction fails at the wait for
> the OM to terminate: the leader logs the skip and all OMs finalize.
> The patch removes this trigger only. Any write whose apply returns
> {{INTERNAL_ERROR}} can have its index flushed before the process exits. It is
> then not applied again after restart. For {{CompleteFinalizeUpgrade}} that
> now needs an I/O failure in an upgrade action or in the VERSION file write.
> Two related fixes in {{QuotaRepairUpgradeAction}}, filed separately and
> linked from there, change the same few lines of
> {{QuotaRepairUpgradeAction.execute}}, so whichever lands later needs a
> trivial rebase.
> Found by TLA+ model checking and code review of the OM upgrade finalization
> paths under HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
