Siyao Meng created HDDS-17237:
---------------------------------
Summary: Tenant create on a Kerberos cluster overwrites the
creating admin's volume list, so their existing volumes disappear from
listVolumes
Key: HDDS-17237
URL: https://issues.apache.org/jira/browse/HDDS-17237
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-use-volume-owner-as-user-key-on-tenant-create.patch,
TestBugCR3TenantCreateOwnerVolumeList.java
h3. Mechanism
{{OMTenantCreateRequest.preExecute}} sets the owner of the tenant volume to the
caller's short name ({{ugi.getShortUserName()}}).
{{OMTenantCreateRequest.validateAndUpdateCache}} instead uses
{{UserInfo.userName}} as the owner, which {{OMClientRequest.getUserInfo}} fills
with {{ugi.getUserName()}}, the full principal under Kerberos (for example
{{[email protected]}}). It takes the user lock, reads the owner's volume list
and adds the updated list to the user table cache under that principal.
{{OMTenantCreateResponse.addToDBBatch}} then writes the list under
{{getUserKey(omVolumeArgs.getOwnerName())}}, the short name.
Every other request that writes the user table ({{OMVolumeCreateRequest}},
{{OMVolumeDeleteRequest}}, {{OMVolumeSetOwnerRequest}}) keys it by the volume
owner, which {{RpcClient.createVolume}} sets to the short name. So the list
read under the full principal is normally empty, the computed list is just the
tenant volume, and the flush replaces the owner's real list with it.
h3. Trigger
# On a Kerberos cluster, {{[email protected]}} (mapped to {{alice}} by
auth_to_local) owns at least one volume, for example {{vol1}} or the volume of
a tenant she created earlier.
# alice, an OM admin, runs {{ozone tenant create tenant1}}.
S3 multi-tenancy can only be enabled with Kerberos, so every tenant create in a
real deployment runs with a full principal. Without Kerberos (user name equals
short name) both keys match and the result is correct.
h3. Impact
* After the flush the user table row {{alice}} is {{[tenant1]}}.
{{listVolumes}} for alice, which serves {{OzoneManager.listVolumeByUser}} and
{{ozone sh volume list}} for the user, no longer returns {{vol1}}. The volume
still exists, is still owned by alice and its data is untouched.
* The per-user volume count ({{ozone.om.user.max.volume}}) reads the same row,
so it undercounts from then on, and it already undercounts in the check during
tenant create itself.
* The row is permanent. No request rebuilds the user table from the volume
table, later volume creates by alice append to the truncated list, and each
further tenant create by alice truncates it again, so an admin who created
several tenants sees only the last tenant volume.
* If the owner's list later becomes empty, for example after the listed tenant
volume is deleted, the row is removed. Deleting a volume that was dropped from
the list, or changing its owner, then fails with {{USER_NOT_FOUND}}
({{OMVolumeRequest.delVolumeFromOwnerList}}), and the volume stays. Creating
any volume for the owner first makes both work again.
h3. Reproduction
PASS on unmodified source when the defect is present.
[^TestBugCR3TenantCreateOwnerVolumeList.java] ({{hadoop-ozone/ozone-manager}};
target path and Maven command in the class comment) sets the RPC caller to a
remote UGI {{[email protected]}} with an auth_to_local rule mapping it to
{{alice}}, as the RPC server does under Kerberos. Each request runs through the
real {{preExecute}}, {{validateAndUpdateCache}}, response {{checkAndUpdateDB}},
a batch commit and a cache cleanup, and the test waits until the user table
cache is evicted, as it is on an OM by the next command. alice's row is read
from RocksDB.
* alice owns {{vol1}} and creates {{tenant1}}: her row is {{[tenant1]}}. After
{{tenant2}} it is {{[tenant2]}}. Setting the owner of {{vol1}} to {{bob}} and
back to {{alice}} gives {{[tenant2, vol1]}}.
* alice owns {{vol1}}, creates {{tenant1}}, deletes the tenant and its volume
(both OK): her row is gone. Deleting {{vol1}} and setting its owner to {{bob}}
both fail with {{USER_NOT_FOUND User not found: alice}}, and {{vol1}} is still
in the volume table. After creating {{vol2}} for alice, deleting {{vol1}} is OK.
* Control with caller {{alice}}: the same steps keep {{[vol1, tenant1,
tenant2]}} and every delete is OK.
With the patch the two Kerberos tests fail at their first check (alice's row is
{{[vol1, tenant1]}}, and {{[vol1]}} after the tenant volume is deleted) and the
control passes. Not run on a Kerberos cluster.
h3. Patch
[^CR-3-use-volume-owner-as-user-key-on-tenant-create.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
41d95dd3286.
{{validateAndUpdateCache}} takes the owner from the {{VolumeInfo}} built in
{{preExecute}} instead of {{UserInfo.userName}}, so the user lock, the read,
the cache entry and the DB write use the same key as the response and the
volume requests. Audit logging still uses {{UserInfo}}. Lists that were already
overwritten are not repaired. A dropped volume can be put back by setting its
owner to another user and back, or, if the owner's row is gone, after creating
any volume for that owner. During a rolling upgrade an unfixed OM still writes
the truncated list for the same entry.
Covered by {{testValidateAndUpdateCacheAddsVolumeToOwnerVolumeList}} in the
existing {{TestOMTenantCreateRequest}}, parameterized with a short name
(control, passes before and after) and a full principal in {{UserInfo}},
checking that the cache holds the owner's list only under the owner key, and
the list in the cache and in RocksDB after the response is written. Without the
change the principal case fails because the cache entry is under
{{[email protected]}}. With it the tenant request suites,
{{TestOMMultiTenantManager}}, {{TestOMMultiTenantManagerImpl}} and the volume
request suites, including volume delete and set owner (59 tests), and the mini
cluster suites {{TestMultiTenantVolume}} and {{TestOzoneTenantShell}} (11
tests, without Kerberos, run before the test was extended) pass, checkstyle is
clean.
Found by code review of the OM volume, bucket and snapshot lifecycle paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
