Siyao Meng created HDDS-16858:
---------------------------------
Summary: Tenant requests release the authorizer write lock only in
their apply, so a request that is not applied on this OM leaves the lock held
and later tenant operations time out
Key: HDDS-16858
URL: https://issues.apache.org/jira/browse/HDDS-16858
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch
h3. Mechanism
Each of the six tenant requests ({{OMTenantCreateRequest}},
{{OMTenantDeleteRequest}}, {{OMTenantAssignUserAccessIdRequest}},
{{OMTenantRevokeUserAccessIdRequest}}, {{OMTenantAssignAdminRequest}},
{{OMTenantRevokeAdminRequest}}) takes the OM wide authorizer write lock in
{{preExecute}} through {{AuthorizerLock.tryWriteLockInOMRequest}}, changes
Ranger, and releases the lock only in the {{finally}} of its
{{validateAndUpdateCache}}. {{AuthorizerLockImpl}} records a single stamp and
holder thread for this lock.
{{OMExecutionFlow.submitExecutionToRatis}} has no step that releases the lock
when the request is not applied on this OM. When
{{OzoneManagerRatisServer.submitRequest}} throws {{ServiceException}} (for
example from {{NotLeaderException}}, {{LeaderSteppingDownException}} or
{{ResourceUnavailableException}}), it calls nothing, and when it returns a
failed response it calls only {{OMClientRequest.handleRequestFailure}}, which
is empty. The tenant requests stopped overriding {{handleRequestFailure}} when
the lock was introduced in HDDS-6701.
The OM is then left with the stamp recorded and no tenant request in flight.
Each later tenant request on that OM fails in {{preExecute}} with {{TIMEOUT}}
("Timed out acquiring authorizer write lock. Another multi-tenancy request is
in-progress. Try again later"), and the Ranger background sync cannot take its
read lock either (from {{OMRangerBGSyncService}}, not run). Because
{{unlockWriteInOMRequest}} releases whatever stamp is recorded, the lock is
freed only when this OM applies a tenant log entry or restarts.
h3. Impact
* Tenant operations (create, delete, user assign and revoke, admin assign and
revoke) fail with {{TIMEOUT}} on the affected OM while it is leader, until it
applies a tenant log entry or restarts.
* The Ranger background sync makes no progress on that OM in the meantime (code
reading, not run).
* No data loss. Key, bucket and S3 operations do not use this lock.
h3. Workaround
Move leadership to another OM ({{ozone admin om transfer}}) and run a tenant
operation there that succeeds. Once the affected OM applies that entry, its
lock is released. Restarting the affected OM also releases it.
h3. Patch
[^MC-1-release-tenant-authorizer-lock-on-ratis-submit-failure.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
The lock is released whatever the submit outcome. {{OMExecutionFlow}} calls a
new hook, {{OMClientRequest.afterSubmit}}, in a {{finally}} after the submit to
Ratis. It is a no op by default, and the six tenant requests override it to
call the new {{AuthorizerLock.unlockWriteInOMRequestIfHeldByCurrentThread}},
which releases the lock only if the current handler thread is its recorded
holder. If the request was applied, its apply has already cleared the holder
before the reply was completed, so the call does nothing and a lock taken
meanwhile by the next tenant request is left alone. Non tenant requests are not
affected, and there is no protocol, config or apply change.
The issue has no standalone reproduction. The regression tests are in the
existing {{TestOMTenantDeleteRequest}}:
{{lockNotHeldAfterSubmitWhateverTheOutcome}} (each submit outcome),
{{responseAfterApplyKeepsLockOfNextRequest}} (the request was applied and the
next request took the lock before the response was returned) and
{{tenantRequestReleasesOnlyOwnLockAfterSubmit}} (each of the six tenant
requests). Without the change the first and third fail because the lock is
still held; the second guards against an unconditional unlock. With it the
tenant and S3 secret request suites, {{TestOMExecutionFlow}},
{{TestAuthorizerLockImpl}}, the multi tenant manager suites and the OM Ratis
suites (165 tests) pass, as do {{TestMultiTenantVolume}} and
{{TestOzoneTenantShell}} on a mini cluster (11 tests). Checkstyle is clean.
HDDS-7184 reported the same symptom ("Timed out acquiring authorizer write
lock" with no multi-tenancy request in progress) and was resolved in 2023 as
probably fixed by HDDS-7755, after the failing test stopped failing. This lock
leak is another way to reach that symptom.
Suggested priority: Major. Tenant operations stop on one OM until it applies a
tenant entry from another leader or restarts. No data is lost and the key and
S3 paths are not affected.
Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger
sync paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests before filing; no existing issue covers it. Related:
HDDS-6701 (introduced the lock and removed the tenant {{handleRequestFailure}}
overrides), HDDS-7184 (same symptom, resolved). The attached patch is a
proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
