[
https://issues.apache.org/jira/browse/HDDS-16855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16855:
------------------------------
Attachment: TestBugMC6OrphanAdminRole.java
> Tenant create that fails in Ranger after creating the admin role leaves the
> role behind, so the tenant can never be created again
> ---------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16855
> URL: https://issues.apache.org/jira/browse/HDDS-16855
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-6-undo-ranger-changes-of-failed-tenant-create.patch,
> TestBugMC6OrphanAdminRole.java
>
>
> h3. Mechanism
> {{OMMultiTenantManagerImpl.AuthorizerOp.createTenant}} makes four Ranger
> calls in order: create the admin role, create the user role (with the admin
> role as delegated admin), create the VolumeAccess policy, create the
> BucketAccess policy. When a later call fails, the catch block only wraps the
> exception into {{TENANT_AUTHORIZER_ERROR}}. Nothing created so far is
> removed, although the Javadoc of the method says "In case of failure: Undo
> all Ranger State", and {{OMTenantCreateRequest.preExecute}} relies on the
> Ranger background sync to "clean up any leftover policies and roles".
> The sync cannot do that for roles without policies.
> {{OMRangerBGSyncService.loadAllPoliciesAndRoleNamesFromRanger}} learns about
> Ranger roles only from the role ACLs of the policies labeled {{OzoneTenant}}.
> A role that no such policy refers to is never seen, never deleted, and the
> sync still records the Ranger service version as in sync. {{createRole}}
> fails when a role with the same name exists (HDDS-15982 kept creation fail
> fast and made only deletes idempotent), so every later create of that tenant
> stops at the first {{createRole}}.
> Related to HDDS-16188 (open, assigned), which covers recovery of tenant
> creation in general. This issue covers only undoing the Ranger changes of a
> failed create and does not resolve HDDS-16188: a create that timed out after
> Ranger created the entity, and reconciliation with entities that already
> exist, stay open there.
> h3. Trigger
> # {{ozone tenant create tenant1}}. Ranger creates {{tenant1-AdminRole}}, then
> the create of {{tenant1-UserRole}} fails once (HTTP 5xx, timeout, Ranger
> restart).
> # The request fails with {{TENANT_AUTHORIZER_ERROR}} in {{preExecute}}.
> Nothing reaches Ratis, OM has no tenant, Ranger keeps the admin role.
> # Any number of sync runs leave the role in place. {{ozone tenant delete
> tenant1}} fails with {{TENANT_NOT_FOUND}} (argued from
> {{OMTenantDeleteRequest.preExecute}}, not run).
> # Every retry of {{ozone tenant create tenant1}} fails with
> {{TENANT_AUTHORIZER_ERROR}}, cause "Role already exists".
> A failure of the VolumeAccess policy create leaves both roles behind in the
> same way. A failure of the BucketAccess policy create leaves the VolumeAccess
> policy, which refers to both roles, so the sync does clean that case up.
> h3. Impact
> The tenant id cannot be created again until an operator deletes the leftover
> roles in Ranger by hand. No data is lost, and the leftover roles carry no
> users and are not referenced by any policy, so they grant nothing. Creating a
> tenant requires cluster admin privilege.
> h3. Reproduction
> PASS, deterministic, unmodified source. [^TestBugMC6OrphanAdminRole.java]
> ({{hadoop-ozone/ozone-manager}}) runs the real
> {{OMTenantCreateRequest.preExecute}}, the real {{AuthorizerOp}}, the real
> {{OMRangerBGSyncService.triggerRangerSyncOnce}} and a real OM DB. Ranger is
> the in-memory controller that OM uses with
> {{ozone.om.tenant.dev.skip.ranger}}, wrapped in a Mockito spy that fails the
> first create of the user role. Output on the unmodified source:
> {noformat}
> create #1: TENANT_AUTHORIZER_ERROR; admin role in Ranger: true, user role in
> Ranger: false, tenant in OM DB: false
> sync run 1: completed true, admin role in Ranger: true
> sync run 2: completed true, admin role in Ranger: true
> retry 1: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
> retry 2: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
> {noformat}
> A passing test means the defect is present. Not run against a real Ranger.
> h3. Suggested fix
> Undo in {{createTenant}} what this call created, as its Javadoc says. Roles
> have to be deleted in reverse order of creation: Ranger refuses to delete a
> role that a policy, another role or a security zone refers to
> ({{RoleDBStore.ensureRoleDeleteAllowed}}), and the user role names the admin
> role as delegated admin. Only objects whose create returned successfully are
> removed, so a role with the same name that existed before (another cluster
> sharing the Ranger, a manual role, or a concurrent duplicate create) is never
> deleted. Two alternatives were rejected: letting the sync find tenant roles
> by name or description is unsafe because Ranger roles are not scoped to a
> service, so the sync of one cluster could delete the tenant roles of another
> cluster using the same Ranger; adopting an existing role in {{createTenant}}
> needs the state aware validation that HDDS-16188 is about.
> The undo narrows the problem but does not remove every case: when the undo
> deletes fail too, or when a create timed out on the client but succeeded in
> Ranger, roles can still be left behind. For example, if the user role create
> times out but succeeds, Ranger refuses the undo's delete of the admin role,
> so both roles stay, as without the patch, and the refused delete is logged at
> WARN.
> h3. Patch
> [^MC-6-undo-ranger-changes-of-failed-tenant-create.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> On failure, {{createTenant}} now deletes the policies and roles it has
> created, best effort, policies first and then the user role before the admin
> role, logging each delete that fails. The deletes are the idempotent ones
> from HDDS-15982. The success path is unchanged. The comment in
> {{OMTenantCreateRequest.preExecute}} that sends all leftovers to the sync is
> updated accordingly.
> Covered by two tests in the existing {{TestOMMultiTenantManagerImpl}}.
> {{testCreateTenantUndoesRangerChangesOnFailure}} fails the user role create,
> then the BucketAccess policy create, checks that nothing is left in Ranger,
> and that the retry succeeds. Without the change it fails with
> "tenant2-AdminRole left in Ranger".
> {{testCreateTenantKeepsExistingRangerRoleOnFailure}} starts with a user role
> that existed before the call: the create fails at the user role, the undo
> deletes the admin role it created and keeps the existing user role. Without
> the change it also fails, because the admin role is left. With the change the
> multi-tenancy unit suites in {{hadoop-ozone/ozone-manager}} (35 tests) and
> {{TestOzoneTenantShell}} and {{TestMultiTenantVolume}} on a mini cluster (11
> tests) pass, checkstyle is clean. With the patch the reproduction fails at
> its first check, because the admin role is gone after the failed create.
> Found by TLA+ model checking and code review of the OM multi-tenancy and
> Ranger sync paths under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
