Siyao Meng created HDDS-17407:
---------------------------------
Summary: Revoked STS token cleanup threshold equals the maximum
token lifetime and leaves no margin for clock differences between OMs
Key: HDDS-17407
URL: https://issues.apache.org/jira/browse/HDDS-17407
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-4-sts-revocation-cleanup-clock-margin.patch
h3. Mechanism
{{RevokedSTSTokenCleanupService}} submits a delete for a revocation cutoff once
{{now - cutoff > CLEANUP_THRESHOLD}}, with {{now}} taken from the clock of the
OM that runs the cleanup and {{CLEANUP_THRESHOLD}} equal to
{{S3STSUtils.MAX_DURATION_SECONDS}} (12 hours). A token covered by the cutoff
has a {{creationTime}} before the cutoff and a duration of at most 12 hours, so
its expiry is before cutoff plus 12 hours. With a single clock the cutoff
therefore outlives every token it covers, with no slack.
Token expiry is checked in {{STSSecurityUtil}} against the clock of the OM that
validates the request. When the purging OM's clock is ahead of the validating
OM's, the cutoff can be deleted while a token it covers has not yet expired on
the validating OM's clock.
h3. Trigger
The purging OM's clock is ahead of the validating OM's by more than (cutoff
minus the token's {{creationTime}}) plus the time between cutoff plus 12 hours
and the cleanup run that deletes the cutoff
({{ozone.om.sts.token.cleanup.service.interval}}, 3 hours by default), for a
token whose duration is close to 12 hours.
h3. Impact
* At this commit {{S3DeleteRevokedSTSTokensResponse}} deletes only the DB row.
OMs that keep the table in memory still hold the cached cutoff; an OM whose
table is loaded from the DB after the delete has no cutoff for the key. This
patch should land before or with HDDS-17406, because with that change the purge
also removes the cutoff from the table cache on every OM, not only from the DB.
* {{S3SecurityUtil.isRevokedStsToken}} reads that state, so on such an OM the
revocation check finds no cutoff for tokens the deleted cutoff covered, until
they expire on its clock. The window is bounded by the clock difference.
* No data is lost or changed. STS is not part of a released Ozone version.
h3. Reproduction
The regression test in the attached patch covers it and fails on unmodified
source:
{{TestRevokedSTSTokenCleanupService.keepsCutoffForClockSkewMarginPastMaxTokenLifetime}}
({{hadoop-ozone/ozone-manager}}) runs the cleanup task with one cutoff 12
hours and 1 minute old and one 12 hours and 31 minutes old, and expects only
the second to be submitted. Without the change it fails with {{but some
elements were not expected: ["original-access-key-within-margin"]}}.
No standalone reproduction is attached. Not run: a multi OM cluster with
different clocks.
h3. Patch
[^MC-4-sts-revocation-cleanup-clock-margin.patch], against
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
65320362775.
{{CLEANUP_THRESHOLD}} becomes the maximum token lifetime plus a constant
{{CLOCK_SKEW_MARGIN}} of 30 minutes, so a cutoff is deleted only after every
token it covers has expired on any OM whose clock is less than 30 minutes
behind the purging OM's. The stored values already bound the expiry of every
covered token by cutoff plus 12 hours; the only unknown is the clock
difference, which nothing stored carries, so a margin is the smallest complete
change. 30 minutes is well above the difference between two OMs that each still
authenticate with the KDC (default allowed skew 5 minutes), and twice the fixed
15 minute request skew the S3 Gateway accepts for SigV4
({{AuthorizationV4HeaderParser}}), which is also a constant. The only cost is
that entries stay 30 minutes longer in a small table. The description of
{{ozone.om.sts.token.cleanup.service.interval}} in {{ozone-default.xml}} now
mentions the margin. A configuration key was not added; it can be if operators
need a different margin.
With the patch, {{TestRevokedSTSTokenCleanupService}}, {{TestS3SecurityUtil}},
{{TestS3RevokeSTSTokenRequest}} and {{TestOmMetadataManager}} pass, and
checkstyle on {{hadoop-ozone/ozone-manager}} and {{hadoop-hdds/common}} is
clean.
Found by TLA+ model checking and code review of the OM STS token revocation and
cleanup paths under HDDS-15926, on commit
5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-14094 (introduced the cleanup service and the 12 hour
threshold), HDDS-16110 (keyed the table by revocation cutoff), HDDS-16879
(apache/ozone#11516; the cleanup service reading a replaced metadata manager, a
different defect). The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
