[
https://issues.apache.org/jira/browse/HDDS-17407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17407:
------------------------------
Affects Version/s: 2.3.0
> Revoked STS token cleanup threshold equals the maximum token lifetime and
> leaves no margin for clock differences between OMs
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17407
> URL: https://issues.apache.org/jira/browse/HDDS-17407
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-4-sts-revocation-cleanup-clock-margin.patch
>
>
> h3. Mechanism
> {{RevokedSTSTokenCleanupService}} submits a delete for a revocation cutoff
> once {{now - cutoff > CLEANUP_THRESHOLD}}, with {{now}} taken from the clock
> of the OM that runs the cleanup and {{CLEANUP_THRESHOLD}} equal to
> {{S3STSUtils.MAX_DURATION_SECONDS}} (12 hours). A token covered by the cutoff
> has a {{creationTime}} before the cutoff and a duration of at most 12 hours,
> so its expiry is before cutoff plus 12 hours. With a single clock the cutoff
> therefore outlives every token it covers, with no slack.
> Token expiry is checked in {{STSSecurityUtil}} against the clock of the OM
> that validates the request. When the purging OM's clock is ahead of the
> validating OM's, the cutoff can be deleted while a token it covers has not
> yet expired on the validating OM's clock.
> h3. Trigger
> The purging OM's clock is ahead of the validating OM's by more than (cutoff
> minus the token's {{creationTime}}) plus the time between cutoff plus 12
> hours and the cleanup run that deletes the cutoff
> ({{ozone.om.sts.token.cleanup.service.interval}}, 3 hours by default), for a
> token whose duration is close to 12 hours.
> h3. Impact
> * At this commit {{S3DeleteRevokedSTSTokensResponse}} deletes only the DB
> row. OMs that keep the table in memory still hold the cached cutoff; an OM
> whose table is loaded from the DB after the delete has no cutoff for the key.
> This patch should land before or with HDDS-17406, because with that change
> the purge also removes the cutoff from the table cache on every OM, not only
> from the DB.
> * {{S3SecurityUtil.isRevokedStsToken}} reads that state, so on such an OM the
> revocation check finds no cutoff for tokens the deleted cutoff covered, until
> they expire on its clock. The window is bounded by the clock difference.
> * No data is lost or changed. STS is not part of a released Ozone version.
> h3. Reproduction
> The regression test in the attached patch covers it and fails on unmodified
> source:
> {{TestRevokedSTSTokenCleanupService.keepsCutoffForClockSkewMarginPastMaxTokenLifetime}}
> ({{hadoop-ozone/ozone-manager}}) runs the cleanup task with one cutoff 12
> hours and 1 minute old and one 12 hours and 31 minutes old, and expects only
> the second to be submitted. Without the change it fails with {{but some
> elements were not expected: ["original-access-key-within-margin"]}}.
> No standalone reproduction is attached. Not run: a multi OM cluster with
> different clocks.
> h3. Patch
> [^MC-4-sts-revocation-cleanup-clock-margin.patch], against
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. It also applies to master at
> 65320362775.
> {{CLEANUP_THRESHOLD}} becomes the maximum token lifetime plus a constant
> {{CLOCK_SKEW_MARGIN}} of 30 minutes, so a cutoff is deleted only after every
> token it covers has expired on any OM whose clock is less than 30 minutes
> behind the purging OM's. The stored values already bound the expiry of every
> covered token by cutoff plus 12 hours; the only unknown is the clock
> difference, which nothing stored carries, so a margin is the smallest
> complete change. 30 minutes is well above the difference between two OMs that
> each still authenticate with the KDC (default allowed skew 5 minutes), and
> twice the fixed 15 minute request skew the S3 Gateway accepts for SigV4
> ({{AuthorizationV4HeaderParser}}), which is also a constant. The only cost is
> that entries stay 30 minutes longer in a small table. The description of
> {{ozone.om.sts.token.cleanup.service.interval}} in {{ozone-default.xml}} now
> mentions the margin. A configuration key was not added; it can be if
> operators need a different margin.
> With the patch, {{TestRevokedSTSTokenCleanupService}},
> {{TestS3SecurityUtil}}, {{TestS3RevokeSTSTokenRequest}} and
> {{TestOmMetadataManager}} pass, and checkstyle on
> {{hadoop-ozone/ozone-manager}} and {{hadoop-hdds/common}} is clean.
> Found by TLA+ model checking and code review of the OM STS token revocation
> and cleanup paths under HDDS-15926, on commit
> 5f7111a27ea07ce6b7bf5284a4213ebacb65f2ae. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-14094 (introduced the cleanup service and the 12 hour
> threshold), HDDS-16110 (keyed the table by revocation cutoff), HDDS-16879
> (apache/ozone#11516; the cleanup service reading a replaced metadata manager,
> a different defect). The attached patch is a proposal for review. Generated
> with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
