[
https://issues.apache.org/jira/browse/HDDS-17176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17176:
------------------------------
Attachment: TestBugMC6TokenAcceptedPastMaxDate.java
> Delegation token is still accepted after its max date when the renew interval
> is longer than the max lifetime
> -------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17176
> URL: https://issues.apache.org/jira/browse/HDDS-17176
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-6-reject-delegation-token-past-max-date.patch,
> TestBugMC6TokenAcceptedPastMaxDate.java
>
>
> h3. Mechanism
> When a token is issued, {{OzoneDelegationTokenSecretManager.updateToken}}
> stores {{issueDate + renewInterval}} as its renew date without capping it at
> the token's {{maxDate}}. {{OzoneDelegationTokenSecretManager.validateToken}},
> which {{retrievePassword}} calls for every token login, checks only that
> renew date and never {{maxDate}}. {{renewToken}} does both: it refuses a
> token past {{maxDate}} with {{TOKEN_EXPIRED}} and caps the new expiry with
> {{Math.min(maxDate, ...)}}.
> {{OzoneManager.createDelegationTokenSecretManager}} does not check
> {{ozone.manager.delegation.token.renew-interval}} against
> {{ozone.manager.delegation.token.max-lifetime}}, so OM starts with a renew
> interval longer than the max lifetime without a warning.
> Hadoop's {{AbstractDelegationTokenSecretManager}} and HDFS (3.4.3) have the
> same behaviour and no check of the two values either, although
> {{hdfs-default.xml}} describes max-lifetime as the lifetime for which a token
> is valid. In Ozone, the {{ozone.manager.delegation.token.max-lifetime}}
> description and the startup check added by HDDS-13343 size the secret key
> lifetime on the assumption that a token is not used after max-lifetime.
> h3. Trigger
> # Configure {{ozone.manager.delegation.token.max-lifetime}} below
> {{ozone.manager.delegation.token.renew-interval}}, for example below the 1d
> default renew interval. OM starts normally.
> # Get a delegation token.
> # After its {{maxDate}}, {{renewDelegationToken}} fails with
> {{TOKEN_EXPIRED}}, but a client holding the token still logs in with it until
> {{issueDate + renewInterval}} (or until signature verification fails because
> the secret key is gone, whichever is earlier).
> Default configs (1d renew interval, 7d max lifetime) do not trigger it.
> Lowering max lifetime is the natural way to get there: the startup check from
> HDDS-13343 requires the secret key expiry to cover max lifetime plus rotation
> and scan intervals, so shortening max lifetime is the knob that satisfies it
> while the renew interval stays at 1d. Several in tree secure integration
> tests run with such a setting, for example {{TestSecureOzoneCluster}} (9 s),
> {{TestDelegationToken}} (60 s), {{TestSecretKeysApi}} (1500 ms) and
> {{TestContainerCommandReconciliation}} (300 s), with the renew interval left
> at its default.
> h3. Impact
> A token authenticates past its documented maximum lifetime, up to the
> configured renew interval. The stored renew date is persisted and reloaded on
> restart, and the expired token remover keys on it, so the window lasts until
> that renew date unless the signing secret key is no longer available (for
> example, after an OM restart a token whose key SCM has already dropped as
> expired is not loaded). Only clusters with max lifetime below the renew
> interval are affected.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC6TokenAcceptedPastMaxDate.java] ({{hadoop-ozone/ozone-manager}})
> builds a real secret manager with max lifetime 1 s and renew interval 1 h,
> issues a token with {{createToken}} and registers it with {{updateToken}} as
> {{OMGetDelegationTokenRequest}} does on apply. After {{maxDate}},
> {{renewToken}} fails with {{TOKEN_EXPIRED}} while {{retrievePassword}} still
> returns the token's password. A passing test means the defect is present;
> with the patch the {{retrievePassword}} step throws {{InvalidToken}}. The
> full token login over RPC was not exercised by this test;
> {{retrievePassword}} is the method the RPC server's SASL handler calls for it.
> h3. Patch
> [^MC-6-reject-delegation-token-past-max-date.patch], against
> b94df213f20ec0d675f079b57a4566e4e64e4060. It also applies to master at
> df42f7f60a2.
> {{validateToken}} now rejects a token whose {{maxDate}} has passed, before
> the renew date check. This also covers tokens issued before the upgrade,
> which keep their uncapped renew date in the store; capping in {{updateToken}}
> alone would not. The patch deliberately does not cap the stored renew date:
> that value is written when the request is applied, so OMs of different
> versions would store different values during a rolling upgrade, and it would
> not help tokens already stored. A token past its {{maxDate}} therefore stays
> in the token table, unusable, until its stored renew date, when the remover
> deletes it as today. With renew interval <= max lifetime the stored renew
> date never exceeds {{maxDate}}, so nothing changes for those configs.
> Rejecting such a config at startup was not chosen because OMs that start
> today would then refuse to start; a startup warning could be added
> separately. The {{ozone.manager.delegation.token.max-lifetime}} description
> in {{ozone-default.xml}} now says the token is not renewed or accepted after
> that interval.
> Covered by {{testRetrievePasswordFailureMaxTime}} in the existing
> {{TestOzoneDelegationTokenSecretManager}}, which checks both a live token and
> one reloaded from the store after a restart. Without the change it fails
> because {{retrievePassword}} returns the password. With it the secret
> manager, token identifier and get delegation token request and response
> suites (30 tests) and the secure mini cluster suite {{TestDelegationToken}}
> pass, checkstyle is clean.
> Found by TLA+ model checking and code review of the OM delegation token
> secret manager and request paths under HDDS-15926, on commit
> b94df213f20ec0d675f079b57a4566e4e64e4060. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
