[
https://issues.apache.org/jira/browse/HDDS-17179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17179:
------------------------------
Attachment:
CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch
> OM skips the secret key expiry check before symmetric token signing is
> finalized, so a runtime finalize issues tokens that fail early
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17179
> URL: https://issues.apache.org/jira/browse/HDDS-17179
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch,
> TestBugCR2FinalizeSkipsSecretKeyGuard.java
>
>
> h3. Mechanism
> HDDS-13343 added a check to
> {{OzoneManager.createDelegationTokenSecretManager}}: OM refuses to start
> unless {{hdds.secret.key.rotate.duration}} +
> {{ozone.manager.delegation.token.max-lifetime}} +
> {{ozone.manager.delegation.remover.scan.interval}} is at most
> {{hdds.secret.key.expiry.duration}}, so that no SCM secret key expires while
> a token it signed can still be renewed. The check only runs when
> {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is already finalized at construction; an
> OM that is not finalized only checks the certificate grace period.
> {{OzoneDelegationTokenSecretManager.createToken}} checks
> {{isAllowed(DELEGATION_TOKEN_SYMMETRIC_SIGN)}} on every call, and
> finalization ({{finalizeupgrade}}, then {{OMCompleteFinalizeUpgradeRequest}}
> and {{OMVersionManager.finalizeUpgrade}}) does not restart OM and does not
> run the check. So an OM that started before finalization with a configuration
> the check would refuse signs delegation tokens with SCM secret keys from the
> moment finalization completes, and the check only fires at the next OM start.
> h3. Trigger
> # A secure cluster that has not finalized {{DELEGATION_TOKEN_SYMMETRIC_SIGN}}
> (layout 8, added in 2.0.0; for example a cluster upgraded from 1.4.x) runs
> this version with a configuration that breaks the rule. The default
> {{hdds.secret.key.expiry.duration}} became 9d in 2.1.0; before that it was
> 7d, so an {{ozone-site.xml}} that sets it explicitly to the old default (7d
> with a 7d token lifetime and 1d rotation) is enough. With default values the
> rule holds and nothing happens.
> # An administrator runs {{ozone admin om finalizeupgrade}}. OM finalizes
> without a restart.
> # A client gets a delegation token, now signed with the current SCM secret
> key.
> # SCM removes that key when it expires, while the token's max date is still
> in the future (for a token signed just before the next rotation, rotation
> period plus token lifetime minus key expiry, so 1d with the values above).
> # Renewing the token fails ("Tampered/Invalid token", caused by "Secret
> verify key ... not found"), and the token cannot recover. Authenticating with
> it goes through the same signature check in
> {{OzoneDelegationTokenSecretManager.retrievePassword}} (code reading, not
> run).
> # The next OM start fails with the check's {{IllegalArgumentException}}. A
> checkpoint install from the leader fails the same way, see Impact.
> h3. Impact
> Tokens issued between finalization and the next OM restart lose up to the
> last day of their lifetime with the configuration above (computed from the
> formula; the scaled run shows the same effect), which breaks long-running
> jobs that renew their tokens up to the max lifetime. The OM then refuses to
> start until the configuration is fixed. An OM that has finalized also
> terminates when it next installs a checkpoint from the leader:
> {{reloadOMState}} calls {{instantiateServices(true)}}, which runs the same
> check, and {{installCheckpoint}} exits the process when the reload throws
> (code reading, not run). Requires an explicit non-compliant configuration on
> a cluster that finalizes this layout feature at runtime; default
> configurations are not affected.
> Affected versions: 2.1.0 and later (2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1), the
> releases that contain the check from HDDS-13343. 2.0.0 has layout 8 but no
> check at all.
> h3. Reproduction
> PASS on the unmodified source. [^TestBugCR2FinalizeSkipsSecretKeyGuard.java]
> ({{hadoop-ozone/integration-test}}) starts a secure SCM (no datanodes, safe
> mode off) and a secure OM with MiniKdc, with the OM storage initialized at
> {{HBASE_SUPPORT}} (layout 7) through the existing
> {{OMStorage.TESTING_INIT_APPARENT_VERSION_KEY}} hook, and scaled timings
> (rotate 10s, expiry 20s, token max lifetime 45s, remover interval 1s). It
> then calls {{finalizeUpgrade}}, gets a token, waits until SCM has removed the
> signing key, renews the token and restarts the OM. Output on the unmodified
> source:
> {noformat}
> OM started pre-finalized, apparent version HBASE_SUPPORT (7), symmetric DT
> allowed false
> pre-finalize token secretKeyId='null' (RSA signed)
> OM finalized at runtime, apparent version GET_KEY_INFO_REJECTS_OBS (103)
> symmetric token issued=1792500182109 maxDate=1792500227109
> signedBy=f068b740-... keyExpiry=1792500196294 (key expires 30815 ms before
> maxDate)
> SCM removed key f068b740-... at 1792500196962, token maxDate 1792500227109 is
> still 30147 ms away
> (OM log) SCMSecurityException: Secret verify key f068b740-... not found for
> token (...)
> renew of unexpired token rejected at 1792500196983 (30126 ms before maxDate,
> ...): TOKEN_ERROR_OTHER ... Delegation token renewal failed ...
> control token signed by 4ee85a92-... renewed, new expiry 1792500241984
> OM restart after runtime finalization refused: Secret key expiry duration
> hdds.secret.key.expiry.duration should be greater than value of (...)
> {noformat}
> A passing test means the defect is present. With the patch the first OM start
> already fails with the check's message.
> h3. Patch
> [^CR-2-check-secret-key-expiry-before-symmetric-token-finalization.patch],
> against b94df213f20ec0d675f079b57a4566e4e64e4060. It also applies to master
> at df42f7f60a2.
> The secret key check now runs at every OM start, whether or not
> {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} is finalized; the certificate grace
> period check still runs only before finalization. This changes behaviour for
> one setup only: an OM that is not finalized and has a configuration that
> breaks the rule now refuses to start (with the same message as after
> finalization) instead of starting and failing later. The
> {{ozone.manager.delegation.token.max-lifetime}} description in
> {{ozone-default.xml}} now says that a secure Ozone Manager fails to start if
> the rule does not hold. The same check also runs in the checkpoint reload, so
> with the patch such an OM refuses at start instead of exiting during a later
> install.
> HDDS-16958 (open) changes the same check and the same {{ozone-default.xml}}
> description. Each patch applies to b94df213f20ec0d675f079b57a4566e4e64e4060
> on its own; this one does not apply on top of the patch attached to
> HDDS-16958 ({{OzoneManager.java}} and {{ozone-default.xml}} conflict, the
> test applies), so whichever lands second needs a small rebase, and the
> {{ozone-default.xml}} clause here can be dropped if HDDS-16958 lands first.
> Rejected alternatives: checking in
> {{OMFinalizeUpgradeRequestBase.preExecute}} covers only the leader's
> configuration and not an OM that finalizes from a leader snapshot; an upgrade
> action for the layout feature runs during Ratis apply, and if it threw on one
> OM that OM would stay at the previous layout while its peers move on.
> Covered by an extension of
> {{assertSecureOmInitFailsWithInvalidTokenAndSecretKeyConfig}} in the existing
> {{TestSecureOzoneCluster}}: OM initialization with the invalid configuration
> is refused when the OM is not finalized (new) and when it is (existing), and
> an OM that is not finalized still starts with a valid configuration. Without
> the change it fails at the case that is not finalized ({{Expected
> java.lang.IllegalArgumentException to be thrown, but nothing was thrown.}}).
> With it {{TestSecureOzoneCluster}} (9 tests), {{TestDelegationToken}},
> {{TestSecureOzoneManager}} and {{TestOMUpgradeFinalization}} (7 tests) on
> mini clusters and the OM unit suites for the finalize requests, the finalize
> service, the delegation token secret manager and the bucket and key managers
> (83 tests) pass, and checkstyle is clean.
> h3. Release note
> A secure Ozone Manager that has not finalized
> {{DELEGATION_TOKEN_SYMMETRIC_SIGN}} now refuses to start if
> {{hdds.secret.key.rotate.duration}} +
> {{ozone.manager.delegation.token.max-lifetime}} +
> {{ozone.manager.delegation.remover.scan.interval}} exceeds
> {{hdds.secret.key.expiry.duration}}, as a finalized one already did. This
> hits secure clusters upgraded from 1.4.x or older and not yet finalized whose
> {{ozone-site.xml}} sets the expiry explicitly (for example to the old 7d
> default) or raises the token max lifetime or rotate duration; raise
> {{hdds.secret.key.expiry.duration}} on SCM and OM (9d covers the defaults)
> before upgrading.
> Found by code review of the OM delegation token secret manager and request
> paths, as part of the TLA+ verification effort under HDDS-15926, on commit
> b94df213f20ec0d675f079b57a4566e4e64e4060. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
