Siyao Meng created HDDS-17199:
---------------------------------
Summary: OMTenantCreateRequest does not pass the requested volume
to TenantOp.createTenant, so the default tenant policies do not cover the
tenant volume
Key: HDDS-17199
URL: https://issues.apache.org/jira/browse/HDDS-17199
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-create-tenant-policies-on-requested-volume.patch
h3. Mechanism
{{OMTenantCreateRequest.preExecute}} takes the volume name from the request,
and the apply creates that volume and records it as the tenant volume in
{{OmDBTenantState}}. The two default policies of the tenant
({{<tenant>-VolumeAccess}} and {{<tenant>-BucketAccess}}) are created by
{{TenantOp.createTenant(tenantId, userRoleName, adminRoleName)}}, which has no
volume argument. {{OMMultiTenantManagerImpl.AuthorizerOp.createTenant}}
therefore builds {{new OzoneTenant(tenantId)}}, whose
{{SingleVolumeTenantNamespace(id)}} uses the tenant id as the volume, so both
default policies name a volume equal to the tenant id.
When the requested volume name differs from the tenant id, the tenant state
records one volume and the two default policies name another.
{{OMRangerBGSyncService}} compares the default policies with the OM DB by name
only, so it finds both and leaves them as they are. Its recovery path for a
missing default policy ({{attemptToCreateDefaultPolicy}}) builds the policy for
the recorded tenant volume, so create and recovery disagree.
h3. Trigger
# An {{OMTenantCreateRequest}} reaches {{preExecute}} on the leader with multi
tenancy enabled and a resolved {{volumeName}} that differs from {{tenantId}}.
# {{AuthorizerOp.createTenant}} creates the tenant roles and the two default
policies in the authorizer.
The state after this is a tenant whose {{OmDBTenantState}} names one volume and
whose two default policies name another. Exercised with
{{InMemoryMultiTenantAccessController}}; the Ranger client implementation was
not exercised.
h3. Impact
* The tenant volume has no default policy, so the tenant's users lack the
default grants on their own volume.
* The background sync does not correct it.
h3. Reproduction
There is no standalone reproduction for this issue; the regression tests are in
the attached patch.
*
{{TestOMMultiTenantManagerImpl.testCreateTenantDefaultPoliciesOnTenantVolume}},
parameterized over a volume equal to the tenant name and a volume different
from it, driving the real {{AuthorizerOp}} against the in memory access
controller and asserting the volume of both default policies. With the new
signature in place but the old {{new OzoneTenant(tenantId)}}, the second case
fails with {{["tenant4"]}} instead of {{["customvol"]}}.
*
{{TestOMTenantCreateRequest.testCreateTenantPassesRequestedVolumeToTenantOps}},
which asserts that both tenant op calls of a create request with a distinct
volume name carry that volume. With the request passing the tenant id to the
authorizer, it fails.
h3. Patch
[^MC-4-create-tenant-policies-on-requested-volume.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to apache master at
1753288f333, alone or together with the patch of HDDS-17198.
{{TenantOp.createTenant}} takes the tenant volume name as a new second
argument, and both implementations follow: {{AuthorizerOp.createTenant}} builds
{{new OzoneTenant(tenantId, tenantVolumeName)}}, and {{CacheOp.createTenant}}
accepts the argument. {{OzoneTenant}} gains a two argument constructor that
builds {{new SingleVolumeTenantNamespace(id, volumeName)}}, while the one
argument constructor delegates to it with the id as the volume name.
{{OMTenantCreateRequest}} passes the {{volumeName}} it resolved to both the
authorizer op in {{preExecute}} and the cache op in {{validateAndUpdateCache}}.
So the two default policies are created on the volume the tenant state records.
Behaviour when the volume name equals the tenant name is unchanged, there is no
proto change, and the authorizer call still runs only on the leader in
{{preExecute}}.
With the patch the tenant, multi tenant manager, access controller, S3 secret
and volume request unit suites (72 tests) and the {{TestMultiTenantVolume}} and
{{TestOzoneTenantShell}} mini cluster suites (11 tests) pass, and checkstyle is
clean. The Ranger client implementation was not exercised.
Two notes for review. For a volume name different from the tenant name, tenant
creation can now fail in {{preExecute}} with {{TENANT_AUTHORIZER_ERROR}} when
the requested volume already has a policy for the same resource, observed with
the in memory access controller; Ranger has a similar rule, not run. And
tenants created before the fix keep their default policies where they are, so
operators can delete both default policies in Ranger and let the background
sync recreate them for the recorded tenant volume, verified with the in memory
access controller, not against Ranger.
Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger
sync paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not a
duplicate: HDDS-6052 (umbrella for making tenant creation with a volume name
different from the tenant name robust and tested), of which this is a concrete
defect, and HDDS-17198 (a separate defect in the same create path). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
