Siyao Meng created HDDS-16790:
---------------------------------
Summary: KeyDeletingService can purge a deleted key in one
PurgeKeys request and carry its size in another, leaving the size in the bucket
usage for good
Key: HDDS-16790
URL: https://issues.apache.org/jira/browse/HDDS-16790
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-keep-deleted-key-and-its-size-in-one-purge-request.patch,
TestBugMC3SplitPurgeLeaksBucketUsage.java
h3. Mechanism
{{KeyDeletingService.submitPurgeKeysRequest}} builds PurgeKeys requests from
block groups, one per version of a {{deletedTable}} entry. For every block
group it adds the name of the entry to the request, but it adds the size to
{{bucketPurgeKeysSize}} only for a committed version. After every block group
it compares the request size with 90% of
{{ozone.om.ratis.log.appender.queue.byte-limit}} and, when the limit is
reached, submits the request and starts a new one. An entry that holds a
committed version and a group of uncommitted blocks can therefore be named
without its size in one request and with its size in the next.
{{OMKeyPurgeRequest}} and {{OMKeyPurgeResponse}} delete the entry by name and
subtract from {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket
only what the same request carries. If the first of the two requests is applied
and the second is not, the entry is gone and nothing subtracts its size any
more.
The size based split (HDDS-13517) and the per bucket size in the request
(HDDS-13764) are both first contained in release 2.1.0 (by history, not run on
older code).
h3. Trigger
# A key is overwritten by a commit that leaves an allocated block uncommitted,
so its {{deletedTable}} entry has the old version and a group of uncommitted
blocks.
# One run of the service collects more than fits into one request. At default
limits (32MB, {{ozone.key.deleting.limit.per.task}} 50000) that needs
{{deletedTable}} keys of about 600 bytes on average, which means key names or
paths of about that length (the average is over every entry of the run,
whatever the bucket layout). It needs less when the per task limit is raised or
the Ratis limit is lowered.
# The cut falls between the two block groups of that entry, with the
uncommitted blocks first. The two groups are close to each other in iteration
order, so only the entries right at a cut are exposed. Which of the two comes
first is about even.
# The next request of the run is not applied.
For step 4 I ran this case: a snapshot of any bucket is created between the two
submits. {{OMKeyPurgeRequest}} then rejects every later request of that run
with {{INVALID_REQUEST}}, because the expected previous snapshot has changed.
No fault is involved. Not run by me: an OM stop or a leader change between the
two submits (argued, the state after the first request is the same). Also
argued, not run: the request that carries the size fails with a
{{ServiceException}}, which the service catches before it submits the next
request, and the request that only names the entry is applied.
h3. Impact
* The bucket keeps the size of the purged entry in {{snapshotUsedBytes}} and
one key in {{snapshotUsedNamespace}}. In the reproduction the bucket is empty,
its deleted table is empty, and both OM and the client still report 805306368
bytes and 1 key in use.
* The leaked amount counts against the quota. With a namespace quota of 1 the
first key written to the empty bucket is rejected with {{QUOTA_EXCEEDED}}.
* Nothing recomputes the counters on its own. A quota repair started by an
admin does (by code reading, not run).
* No data is lost and the blocks are deleted as intended. One incident leaks
the entries that sit at a cut, typically one.
h3. Reproduction
PASS on unmodified source. [^TestBugMC3SplitPurgeLeaksBucketUsage.java]
({{hadoop-ozone/ozone-manager}}) drives one real OM through
{{OzoneManagerProtocol}}, without datanodes. It overwrites 20 keys so that each
leaves one uncommitted block, runs {{KeyDeletingService}} once, and creates a
snapshot of an unrelated bucket through the client API right after the first
applied request that names an entry without its size. It then shows that the
later request for that entry is rejected with {{INVALID_REQUEST}}, that the
deleted table is empty, and the leaked usage and the {{QUOTA_EXCEEDED}} error
described above. To keep the test small the Ratis limit is 64KB and the key
names are 59000 characters long, so every block group fills its own request. A
split at default limits (about 30MB of names in one run) was not run. A passing
test means the defect is present.
h3. Patch
[^MC-3-keep-deleted-key-and-its-size-in-one-purge-request.patch], against
a6332fb59879b80615b9233e953d6813d03ae7a8. It also applies to master at
3a22acfaa21.
{{KeyDeletingService.submitPurgeKeysRequest}} groups the block groups by
deleted key name and adds one {{deletedTable}} entry per step, its name once
and the sizes of all its committed versions together. The size check stays
where it is, so a request can only be cut between entries. An entry that alone
exceeds the limit is sent in a request of its own, as before. The request
format and the way it is applied do not change. One visible side effect:
{{NumKeysPurged}} is counted from the names in the request, so an entry with
two reclaimable versions now counts once instead of twice.
{{NumKeysSentForPurge}} and the deleted key count of the service still count
versions, so the two no longer agree for such entries.
Covered by {{testPurgeKeysRequestKeepsKeyVersionsTogether}} in the existing
{{TestKeyDeletingService}}, which also pins that an entry that is rewritten
instead of purged carries its size in the request that updates it. Without the
change it fails because the overwritten entry is spread over two requests (four
requests instead of three). With it {{TestKeyDeletingService}},
{{TestOMKeyPurgeRequestAndResponse}} and {{TestKeyManagerImpl}} pass (59 tests)
and checkstyle is clean. {{TestKeyPurging}} and
{{TestSnapshotDeletingServiceIntegrationTest}} on a mini cluster (9 tests) pass
in a run that had this patch and a second, unrelated change to
{{OMKeyPurgeRequest}} applied together. With the patch the reproduction fails
at its first assertion, because no request names an entry without its size any
more. Buckets that already leaked are not repaired by the patch.
Found by TLA+ model checking and code review of the OM key deletion service and
key purge paths under HDDS-15926, on commit
a6332fb59879b80615b9233e953d6813d03ae7a8. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-13517 (introduced the size based split of PurgeKeys requests),
HDDS-13764 (introduced the per bucket size in the request), HDDS-16184 (quota
repair of the same counters). The attached patch is a proposal for review.
Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
