Siyao Meng created HDDS-16857:
---------------------------------
Summary: Ranger background sync deletes the default policies of a
tenant whose create is applied but not yet flushed to RocksDB
Key: HDDS-16857
URL: https://issues.apache.org/jira/browse/HDDS-16857
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-5-ranger-sync-keep-policies-of-unflushed-tenants.patch,
TestBugMC5SyncDeletesUnflushedTenantPolicies.java
h3. Mechanism
{{OMRangerBGSyncService.processAllPoliciesFromOMDB}} decides which labeled
Ranger policies to keep by iterating {{getTenantStateTable().iterator()}}. That
iterator reads RocksDB only and skips the table cache. The role members,
{{tenantCache}} and {{OMMultiTenantManagerImpl.getTenantVolumeName}} all see
applied state, including entries still waiting in the double buffer. Every
labeled policy starts on the delete list, and nothing in the sync waits for the
flush.
This gives two symptoms:
* A tenant create that is applied but not flushed is missing from the keep set,
so the sync deletes both default policies of that live tenant
({{<tenant>-VolumeAccess}} and {{<tenant>-BucketAccess}}).
* A tenant delete that is applied but not flushed is still in RocksDB. Its
policies are already gone from Ranger, so the sync tries to recreate them.
{{attemptToCreateDefaultPolicy}} then gets {{TENANT_NOT_FOUND}} from
{{getTenantVolumeName}}, and the whole sync run aborts ("Exception during
Ranger Sync"), so no other tenant is reconciled in that run.
h3. Trigger
# An OM admin creates (or deletes) a tenant. The request updates Ranger in
preExecute, updates the table cache and {{tenantCache}} in
validateAndUpdateCache, and releases the authorizer lock.
# The sync on the leader takes its optimistic read after that and finishes
loading from Ranger before the double buffer has flushed the request.
With normal flush latency the flush lands well before the sync reaches the
policy phase, so this needs a slow flush (argued, not measured). Once the
create is flushed, attempt 2 of the same run or the next run (up to
{{ozone.om.multitenancy.ranger.sync.interval}}, default 10m, later) recreates
the policies. Attempt 2 does not wait for the flush, so if it also runs before
the flush, the policies stay missing until the next run.
h3. Impact
For a tenant that every OM reader sees as existing, Ranger has no default
policies during the window, so its users lose the grants on the tenant volume
and buckets. That access effect is argued; no Ranger plugin ran in the tests. A
pending delete blocks the reconcile of all tenants for the same window. No OM
state is lost, and the sync restores the policies by itself.
h3. Reproduction
[^TestBugMC5SyncDeletesUnflushedTenantPolicies.java]
({{hadoop-ozone/ozone-manager}}). It runs the real {{OMTenantCreateRequest}}
(preExecute and validateAndUpdateCache), the manager's own
{{OMRangerBGSyncService}} and a real {{OmMetadataManagerImpl}}. Ranger is the
in-tree {{InMemoryMultiTenantAccessController}}, and the flush is done by hand.
On the unmodified code it passes: after one sync run the tenant is in
{{tenantCache}} but Ranger has no policies for it, and the policies come back
only after the flush and a second run. The control test (flush before the sync)
keeps them. A passing test means the defect is present.
h3. Patch
[^MC-5-ranger-sync-keep-policies-of-unflushed-tenants.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
{{processAllPoliciesFromOMDB}} builds the keep set from the same view as the
rest of OM. It collects the tenant IDs from {{cacheIterator}} first and then
from {{keyIterator}}, the same order
{{OmMetadataManagerImpl.getMultipartUploadKeys}} uses. It reads each tenant
through the cache and skips a tenant deleted in the cache. Once the double
buffer is flushed, the result is the same as before. A keep set built only from
{{tenantCache}} would also fix this, but it reads no table. After a snapshot
install the previous sync service keeps running on the closed DB (HDDS-16853),
and today it fails on that DB before its first Ranger write. With a
{{tenantCache}} keep set it would no longer fail there, and would go on to
write to Ranger from the tenants of the old DB.
Covered by {{testRangerSyncKeepsPoliciesOfUnflushedTenant}} and
{{testRangerSyncSkipsDeletedUnflushedTenant}} in the existing
{{TestOMMultiTenantManagerImpl}}. Without the change the first finds only the
policies of the flushed tenant, and the second finds the sync run failed. With
it, those two tests and the tenant request, manager and in-memory controller
unit suites pass (25 tests), as do {{TestMultiTenantVolume}} and
{{TestOzoneTenantShell}} (11 tests), and checkstyle is clean. With the patch
the reproduction fails, because the policies are kept.
Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger
sync paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
