fmorg-git opened a new pull request, #11810: URL: https://github.com/apache/ozone/pull/11810
Please describe your PR in detail: * RevokedSTSTokenCleanupService picks the s3RevokedStsTokenTable entries to purge from revokedStsTokenTable.iterator(), which reads RocksDB only (TypedTable.iterator does not consult the table cache), and submits DeleteRevokedSTSTokens with the original access key IDs whose cutoff is older than 12h. The request carries the key IDs only. S3DeleteRevokedSTSTokensRequest.validateAndUpdateCache does not read the current cutoff and writes nothing to the cache, and S3DeleteRevokedSTSTokensResponse.addToDBBatch deletes every listed key from the DB. This has two effects on the table, which is a full cache table: A purge entry deletes whatever cutoff the key has in the DB when the entry is applied, not the one the scan read. When a RevokeSTSToken for the same key that writes a newer cutoff is ordered before the purge entry, the purge removes that newer cutoff from the DB. The table cache keeps it, because the purge writes no cache entry, so an OM that has not rebuilt its cache still reads it, while an OM that rebuilds the cache from the DB (restart, checkpoint install) has no cutoff for that key. Every purged entry stays in the table cache, since FullTableCache only evicts entries whose cache value is null. Cache first reads (getIfExist) and the cache size keep reporting purged cutoffs on every OM until it rebuilds the cache, while the DB has none. S3SecurityUtil.isRevokedStsToken reads this table through the cache (getIfExist). Trigger For the first effect, the leader's cleanup scan reads a cutoff older than 12h for a key, and a RevokeSTSToken for the same key that writes a newer cutoff is ordered before the purge entry. Two states lead there, both reproduced with the real scan and apply code: The newer cutoff is applied on the leader but not yet flushed to RocksDB when the scan creates its iterator. The newer RevokeSTSToken is applied after the scan, but is ordered before the purge entry in the log. The older cutoff must still be in the table (the first cleanup run after it turns 12h old removes it; the default interval is 3h), so this is a race with one cleanup run. The second effect follows from every purge. Impact First effect: after the purge is flushed, the DB of every OM has no row for the key, although the last applied RevokeSTSToken for it wrote a newer cutoff. OMs that rebuild the table cache from the DB have no cutoff for the key; OMs that did not still have the newer one in the cache. Revocation checks for STS tokens of that key created before the newer cutoff read that state and differ between the two groups until those tokens expire (at most 12h). Argued from S3SecurityUtil.isRevokedStsToken, not run on a cluster. Second effect: purged cutoffs stay readable through the cache and counted in the cache size until a cache rebuild, so OMs at the same applied index hold different cache contents. A purged cutoff is at least 12h old and STS tokens live at most 12h, so this does not change any revocation check. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17406 ## How was this patch tested? unit tests, ci workflows -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
