Siyao Meng created HDDS-16859:
---------------------------------
Summary: Ranger background sync pushes the role members loaded at
the start of the attempt instead of the current ones
Key: HDDS-16859
URL: https://issues.apache.org/jira/browse/HDDS-16859
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-ranger-sync-push-current-role-members.patch
h3. Mechanism
Each attempt of {{OMRangerBGSyncService.executeOMDBToRangerSync}} first loads
the Ranger roles and the OM role members
({{OMMultiTenantManagerImpl.getAllRolesFromCache}}).
{{processAllRolesFromOMDB}} then compares them, and for each role that differs
{{pushOMDBRoleToRanger}} takes the write lock and writes the members from that
load to Ranger. Tenant requests change Ranger and the tenant cache under the
same lock, so by the time of the push the loaded members can be older than the
tenant cache.
The {{createRole}} branch of {{processAllRolesFromOMDB}} decides from the same
load. If the tenant is deleted after the load, the branch creates its role
again.
h3. Impact
* Right after a push, the members of the Ranger role can differ from the tenant
cache: Ranger has the members of the load, not those of the last completed
tenant request.
* A role created again for a deleted tenant stays in Ranger. The sync finds
Ranger roles only through the labeled policies, and the deleted tenant has
none, so no later run deletes the role. A later tenant create with the same
name then fails in Ranger on the existing role (code reading, not run against
Ranger).
h3. Reproduction
There is no standalone reproduction. The two regression tests in the attached
patch cover it and fail on unmodified source. Both use
{{InMemoryMultiTenantAccessController}} in place of Ranger.
* {{TestOMMultiTenantManagerImpl.testRangerSyncPushesCurrentRoleMembers}}: the
OM members of a differing role change after the load. Right after the push
Ranger must have the current members. Without the change it has {{["user2",
"seed-user1"]}} instead of {{["seed-user1"]}}.
*
{{TestOMMultiTenantManagerImpl.testRangerSyncDoesNotCreateRoleOfTenantDeletedDuringSync}}:
the tenant is deleted after the sync created its policies again and before it
creates its roles. After the run Ranger must have neither role. Without the
change both roles are there.
h3. Patch
[^CR-1-ranger-sync-push-current-role-members.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
A new private helper {{OMRangerBGSyncService.getCurrentRoleUsers}} reads the
role members from {{getAllRolesFromCache}}, or returns the loaded members for
other {{OMMultiTenantManager}} implementations, as {{loadAllRolesFromOM}} does.
{{pushOMDBRoleToRanger}} calls it under its write lock and writes the members
it returns. The {{createRole}} branch calls it under its own write lock. Both
skip the role when its tenant has been deleted. Without concurrent tenant
requests the sync writes the same data as before.
Not changed: the orphan {{deleteRole}} branch also decides from the load. A
role it deletes by mistake is created again by the next run (code reading, not
run).
With the patch the tenant request, manager and in-memory controller unit suites
pass (25 tests), as do {{TestMultiTenantVolume}} and {{TestOzoneTenantShell}}
(11 tests), and checkstyle on {{hadoop-ozone/ozone-manager}} is clean.
Found by code review of the OM multi-tenancy and Ranger sync paths, as part of
the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not a
duplicate: HDDS-7178 (introduced the optimistic read of the load). The attached
patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
