[
https://issues.apache.org/jira/browse/HDDS-17198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17198:
------------------------------
Attachment: MC-3-reject-tenant-create-on-volume-used-by-another-tenant.patch
> Force creating a tenant on a volume already used by another tenant terminates
> every OM, also on restart
> -------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17198
> URL: https://issues.apache.org/jira/browse/HDDS-17198
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Critical
> Attachments:
> MC-3-reject-tenant-create-on-volume-used-by-another-tenant.patch
>
>
> h3. Mechanism
> With {{forceCreationWhenVolumeExists}} set,
> {{OMTenantCreateRequest.preExecute}} skips the volume check and never looks
> at the reference count of the existing volume, so the request is committed.
> In {{OMTenantCreateRequest.validateAndUpdateCache}} the force path increments
> the {{refCount}} of the existing volume and then runs
> {{Preconditions.checkState(omVolumeArgs.getRefCount() == 1L, "refCount should
> have been set to 1")}}. If the volume already belongs to a tenant, the count
> becomes 2 and an {{IllegalStateException}} is thrown. The request only
> catches {{IOException}} and {{InvalidPathException}}, so the exception
> reaches {{OzoneManagerStateMachine.runCommand}}, which calls
> {{ExitUtils.terminate}}.
> The entry is in the Raft log, so every OM terminates when it applies it, and
> an OM that is restarted replays it and terminates again.
> In the review of HDDS-8994 (apache/ozone#5045, which added the hidden
> {{--force}} option) this case was discussed and the precondition was
> described as what makes several tenants on one volume impossible, so a
> rejection was intended rather than a crash.
> h3. Trigger
> # A cluster admin creates tenant {{tenanta}} (volume {{tenanta}},
> {{refCount}} 1).
> # The admin calls {{ObjectStore.createTenant("tenantb",
> TenantArgs.newBuilder().setVolumeName("tenanta").setForceCreationWhenVolumeExists(true).build())}}.
> {{RpcClient.createTenant}} passes both fields unchanged. The shell cannot do
> it on its own, because {{ozone tenant create --force}} always uses the tenant
> name as the volume name. Two admins force creating two different tenants on
> the same plain volume at the same time also reach it, because both pass the
> volume checks in {{preExecute}}, which run before the authorizer lock is
> taken, before either is applied (covered by the unit test below, not run on a
> cluster).
> h3. Impact
> * Every OM terminates on the committed entry, and the client gets
> {{INTERNAL_ERROR}}.
> * A restarted OM terminates again when it replays the entry, so the OM
> service stays down until the binary is fixed or the entry is removed from the
> log.
> * The request requires cluster admin rights.
> h3. Reproduction
> The issue has no standalone reproduction. The regression test in the attached
> patch covers it (see below).
> h3. Patch
> [^MC-3-reject-tenant-create-on-volume-used-by-another-tenant.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> 1753288f333.
> {{validateAndUpdateCache}} throws {{OMException}} with the existing result
> code {{VOLUME_IS_REFERENCED}} instead of failing the precondition, so the
> request fails and the OM keeps running. This also makes replay of an entry
> that is already in the log harmless once the OMs run the fixed code.
> {{preExecute}} rejects the request with the same code when the existing
> volume has a nonzero {{refCount}}, so in the common case nothing is created
> in Ranger for the rejected tenant. No proto or client change. Requests that
> succeeded before behave the same.
> Covered by {{testForceCreationWhenVolumeIsUsedByAnotherTenant}} in the
> existing {{TestOMTenantCreateRequest}}: two force requests on one plain
> volume are both pre executed, the second apply must return
> {{VOLUME_IS_REFERENCED}} and leave the volume and tenant tables unchanged,
> and a later request must be rejected in {{preExecute}}. Without the change it
> fails with {{IllegalStateException: refCount should have been set to 1}}.
> With it the tenant and volume request suites and
> {{TestOMMultiTenantManagerImpl}} (50 tests) and the {{TestMultiTenantVolume}}
> and {{TestOzoneTenantShell}} mini cluster suites (11 tests) pass, checkstyle
> is clean. Not run: a restart of a fixed OM over a log written by an unfixed
> one.
> Found by TLA+ model checking and code review of the OM multi-tenancy and
> Ranger sync paths under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
