Siyao Meng created HDDS-16823:
---------------------------------
Summary: OMRecoverLeaseRequest calls SCM and generates block
tokens in validateAndUpdateCache, so applying RecoverLease depends on SCM
Key: HDDS-16823
URL: https://issues.apache.org/jira/browse/HDDS-16823
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-move-recover-lease-scm-call-out-of-apply.patch
h3. Mechanism
{{OMRecoverLeaseRequest.validateAndUpdateCache}} runs on the OM state machine
apply thread of every OM. Through {{doWork}} and {{updateBlockInfo}} it calls
{{getContainerWithPipeline}} on SCM, up to three times (last block of the file,
last two blocks of the open file), while holding the bucket write lock. The
pipelines, and the block tokens generated next to them, are only put into the
reply for the recovering client. They are never written to the OM DB.
An {{IOException}} from the SCM call is not an {{OMException}}, so
{{OzoneManagerRatisUtils.exceptionToResponseStatus}} turns it into
{{INTERNAL_ERROR}}, which {{OzoneManagerStateMachine.processResponse}} treats
as an unrecoverable error of the OM. The request is a replicated log entry, so
every OM that applies it makes the same call. {{RecoverLease}} is the only OM
write request that calls SCM from {{validateAndUpdateCache}}. Block allocation
and upgrade finalization call it from {{preExecute}} on the leader.
HDDS-10626 was an earlier OM shutdown from the same method (block token
generation while a {{RecoverLease}} entry was applied) and was fixed by
changing the startup order, which left both calls in the apply path.
h3. Trigger
A {{RecoverLease}} entry is applied while {{getContainerWithPipeline}} fails
with an exception that is not an {{OMException}}.
h3. Impact
* The entry is answered with {{INTERNAL_ERROR}} and the state machine
terminates the OM, on each OM that applies it. The status is shown by the
regression test below. What the state machine does with that status is from
reading {{processResponse}}.
* Without any failure, each OM's single apply thread waits for up to three SCM
calls with the bucket write lock held, so no other write is applied on that OM
in the meantime.
h3. Reproduction
There is no standalone reproduction for this issue. The regression test in the
attached patch covers it: on unmodified source the apply level part of
{{TestOMRecoverLeaseRequest.testScmFailureDoesNotFailApply}} (a mocked SCM
client that throws {{IOException}}) fails with "expected: <OK> but was:
<INTERNAL_ERROR>".
h3. Patch
[^CR-1-move-recover-lease-scm-call-out-of-apply.patch], against
ea69b4a7d9abd040e5259dbcbb7c6ce52eb5d199. It also applies to master at
84f5594a6a2.
{{validateAndUpdateCache}} no longer talks to SCM or generates block tokens.
The leader adds both to the reply in
{{OzoneManagerProtocolServerSideTranslatorPB.internalProcessRequest}}, after
the request has been applied and outside any OM lock, in the same place where
the S3 derived key is already added to a {{CreateKey}} reply. If the SCM call
fails there, the caller gets {{SCM_GET_PIPELINE_EXCEPTION}} (the code
{{KeyManagerImpl}} already uses for a failed pipeline refresh). The file is
then already marked as under recovery, and a later {{recoverLease}} call
continues from there, as a second recovery of the same file does today.
There is no wire or protocol change, and entries already in the log are applied
as before, without the SCM call. Two side effects. The block token is now
issued for the calling user, as it is for block allocation, instead of the OM's
own user (the apply thread has no RPC caller). And the SCM call still blocks
its caller for as long as the SCM client takes, but on the RPC handler thread
of the leader only, not on the apply thread and not under the bucket lock.
Covered by {{testPipelinesAreAddedAfterApply}} and
{{testScmFailureDoesNotFailApply}} in the existing
{{TestOMRecoverLeaseRequest}}. With the patch {{TestOMRecoverLeaseRequest}},
{{TestOpenKeyCleanupService}} and {{TestOMKeyCommitRequestWithFSO}} (48 tests)
pass and checkstyle is clean. On a mini cluster {{TestHSync}} (42),
{{TestSecureOzoneRpcClient}} (241, block tokens enabled) and {{TestOFSWithFSO}}
(73, 9 skipped) pass.
Found by code review of the lease recovery paths for hsynced files, as part of
the TLA+ verification effort under HDDS-15926, on commit
ea69b4a7d9abd040e5259dbcbb7c6ce52eb5d199. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-10626 (earlier shutdown from the same method), HDDS-16767
(wall clock read while the same request is applied). The attached patch is a
proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
