[
https://issues.apache.org/jira/browse/HDDS-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sreeja Chintalapati reassigned HDDS-17457:
------------------------------------------
Assignee: Sreeja Chintalapati
> Bucket creation right after a bucket delete is rejected with TOO_MANY_BUCKETS
> at the bucket limit, because the deleted bucket is still counted
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17457
> URL: https://issues.apache.org/jira/browse/HDDS-17457
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Assignee: Sreeja Chintalapati
> Priority: Minor
> Attachments: CR-1-count-only-live-buckets-towards-bucket-limit.patch,
> TestBugCR1MaxBucketDeleteMarker.java
>
>
> h3. Mechanism
> {{OMBucketCreateRequest.validateMaxBucket}}, called from {{preExecute}} on
> the leader, rejects a create when {{bucketTable.getEstimatedKeyCount()}} is
> at least {{ozone.om.max.buckets}}. The bucket table has a full cache, so
> {{TypedTable.getEstimatedKeyCount}} returns {{FullTableCache.size()}}. A
> deleted bucket stays in that cache as an entry with a null value until the
> delete is flushed and the cache cleanup evicts it (since 2.0.0 the cleanup
> runs every second; in 1.4.x it runs right after the flush, so the window ends
> at the flush), and the DeleteBucket reply is sent after apply, before the
> flush. Until then the deleted bucket still counts toward the limit.
> h3. Trigger
> The cluster has {{ozone.om.max.buckets}} buckets. A client deletes one and
> creates another within about a second (longer when the double buffer is
> slow). No fault injection is needed.
> h3. Impact
> The create fails with {{TOO_MANY_BUCKETS}} ("Cannot create more than N
> buckets") although fewer than N buckets exist. Nothing retries it in the OM,
> and a caller that trusts the error believes the limit is reached. A retry
> after the eviction succeeds. No state is affected.
> h3. Reproduction
> PASS on unmodified source, deterministic, about 14 seconds.
> [^TestBugCR1MaxBucketDeleteMarker.java] ({{hadoop-ozone/ozone-manager}})
> starts one real OM with single node Ratis ({{OmTestManagers}}) and uses only
> the client API with {{ozone.om.max.buckets}} = 3. A create at the limit is
> rejected as intended. With the double buffer held by its pause hook, so that
> the window does not depend on timing, it deletes one bucket, {{listBuckets}}
> returns 2 buckets, and the next create is rejected with {{TOO_MANY_BUCKETS}}.
> After the delete is flushed and evicted, the same create succeeds. A passing
> test means the defect is present.
> h3. Patch
> [^CR-1-count-only-live-buckets-towards-bucket-limit.patch], against
> a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
> b6d0181075e.
> {{validateMaxBucket}} keeps the cheap count and, only when that count reaches
> the limit, counts the live buckets (non-null values) in the bucket table
> cache, which holds every bucket. Below the limit nothing changes. At the
> limit, each create request walks the in-memory bucket cache once, without
> taking a lock (estimated at about 1 ms for 100000 buckets from a pass over a
> plain skip list of that size, not measured in an OM). The check runs only on
> the leader and is not part of the replicated state, so followers, apply and
> mixed-version OMs are not affected. {{FullTableCache.size()}} is unchanged
> because it also serves the cache size metric.
> Covered by {{preExecuteBucketAfterDeleteAtMaxLimit}} in the existing
> {{TestOMBucketCreateRequest}} (also run by
> {{TestOMBucketCreateRequestWithFSO}}): at a limit of 1, a create after a
> delete must pass, and a further create must still be rejected. Without the
> change the first create fails with "Cannot create more than 1 buckets". With
> it, the bucket request suites and {{TestBucketManagerImpl}},
> {{TestOmMetadataManager}} and {{TestOzoneManagerRequestHandler}} (161 tests)
> pass, and checkstyle is clean. With the patch the reproduction fails, because
> the create in the window succeeds.
> Found by code review of the OM table cache, double buffer flush and startup
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
