Priyesh Karatha created HDDS-17503:
--------------------------------------
Summary: Recon: report OM pending deletion as purgeable vs
snapshot-held using live OM data
Key: HDDS-17503
URL: https://issues.apache.org/jira/browse/HDDS-17503
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Recon
Reporter: Priyesh Karatha
Assignee: Priyesh Karatha
/api/v1/pendingDeletion?component=om currently builds its numbers from Recon's
own copy of the OM DB (ReconGlobalMetricsService.calculatePendingSizes()). That
data:
- lags behind OM
- can't tell purgeable data apart from data still held by snapshots
OM now offers GetBucketDeletedBytes, which classifies deleted data per bucket
using the snapshot reclaimable filters.
Proposed change:
- New BucketDeletedBytesMetricsService: a singleton that calls
GetBucketDeletedBytes once per bucket in Recon's bucket table, using a thread
pool. One BucketDeletedBytesCollectionTask runs per bucket, and the service
adds up:
- purgeable bytes and snapshot-held bytes
- purgeable key and directory counts
- purgeable key size and purgeable directory size
- Rate limiting and timeout: collection runs asynchronously and is rate
limited. New configs:
- ozone.recon.bucket.deleted.bytes.collection.minimum.api.delay (default 30s)
- ozone.recon.bucket.deleted.bytes.collection.timeout (default 10m)
- ozone.recon.bucket.deleted.bytes.collection.thread.count
- Failure handling:
- A bucket that fails is counted in totalBucketQueriesFailed and left out of
the totals.
- If the collection run itself fails, the status is FAILED.
- If Recon's OM DB copy isn't loaded yet, the request fails and the next
request retries, instead of staying stuck in NOT_STARTED.
- /api/v1/pendingDeletion?component=om:
- Returns 202 with \{status, message} while the status is NOT_STARTED,
IN_PROGRESS or FAILED.
- Returns 200 once FINISHED, with totalPurgeableSize,
totalSnapshotTrappedSize, totalPurgeableKeys, totalPurgeableDirs,
totalPurgeableKeySize, totalPurgeableDirSize, totalBucketsQueried and
totalBucketQueriesFailed.
- Older OM: if OM doesn't report the key/dir split, totalPurgeableKeySize and
totalPurgeableDirSize are -1.
- /api/v1/buckets: adds per-bucket trappedBytes and trappedKeys (-1 until they
are collected) and trappedBytesCollectionStatus.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
