[
https://issues.apache.org/jira/browse/HDDS-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16879:
------------------------------
Attachment: CR-2-sts-token-cleanup-read-current-metadata-manager.patch
> RevokedSTSTokenCleanupService keeps the metadata manager closed by a
> checkpoint install, so every cleanup run fails while that OM is leader
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16879
> URL: https://issues.apache.org/jira/browse/HDDS-16879
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> CR-2-sts-token-cleanup-read-current-metadata-manager.patch,
> TestBugCR2StsCleanupClosedMetadataManager.java
>
>
> h3. Mechanism
> {{RevokedSTSTokenCleanupService}} stores
> {{ozoneManager.getMetadataManager()}} in a final field in its constructor,
> and the service is created once in {{OzoneManager.start}}.
> {{OzoneManager.installCheckpoint}} stops the old metadata manager, which
> closes its store, and {{reloadOMState}} creates a new
> {{OmMetadataManagerImpl}}. The service is not stopped, recreated or updated
> on that path, so each later run iterates {{s3RevokedStsTokenTable}} of the
> closed store.
> The service is not in a release yet (HDDS-14094), so only master is affected.
> The OM checkpoint servlets have the same problem, see HDDS-16878.
> h3. Trigger
> # An OM installs a checkpoint from the leader through
> {{installSnapshotFromLeader}}.
> # That OM becomes leader (election or leadership transfer). The service only
> runs on the leader.
> h3. Impact
> * While that OM is leader, every run fails with {{RocksDatabaseException:
> Rocks Database is closed}}, logs "Failure while scanning
> s3RevokedStsTokenTable" at ERROR with a stack trace, and deletes nothing,
> until the OM restarts. With the default
> {{ozone.om.sts.token.cleanup.service.interval}} of 3h that is one ERROR every
> 3 hours. The scan fails on an empty table too, so the ERROR appears whether
> or not STS is used.
> * Expired revocation entries stay in the table while that OM leads. They
> belong to tokens past their maximum lifetime, and revocation checks read the
> current table, so no authentication result changes. Another OM that has not
> installed a checkpoint since it started cleans up the backlog once it becomes
> leader (code reading).
> h3. Reproduction
> PASS on the unmodified source, deterministic.
> [^TestBugCR2StsCleanupClosedMetadataManager.java]
> ({{hadoop-ozone/integration-test}}) starts three OMs with one held back and
> the cleanup interval set to 1s, advances the leader past the Ratis log purge
> gap, starts the late OM so that Ratis installs a checkpoint on it, and
> transfers leadership to it. One revocation entry 13 hours old is then written
> directly into the new leader's table, because a real revoke cannot be back
> dated; it only makes the missed cleanup visible. Output:
> {noformat}
> After omNode-3 installed a checkpoint and became leader: scan failures logged
> = 4, 'Rocks Database is closed' in log = true, expired entry still present =
> true
> ERROR service.RevokedSTSTokenCleanupService
> (RevokedSTSTokenCleanupService.java:call(187)) - Failure while scanning
> s3RevokedStsTokenTable. It will be retried in the next interval
> org.apache.hadoop.hdds.utils.db.RocksDatabaseException: Rocks Database is
> closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:417)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:809)
> {noformat}
> A passing test means the defect is present. With the patch no scan fails, the
> service logs "Found and removed 1 revoked STS token entries." and the test
> fails at its first defect assertion.
> h3. Patch
> [^CR-2-sts-token-cleanup-read-current-metadata-manager.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> 6818707698c.
> The service no longer keeps the metadata manager and reads
> {{ozoneManager.getMetadataManager()}} at the start of each run, as
> {{OMUpgradeFinalizeService}} already does. Nothing changes for an OM that
> never installed a checkpoint, and no lifecycle code is added to the install
> path.
> Covered by the new
> {{TestRevokedSTSTokenCleanupService#scansCurrentMetadataManagerAfterItIsReplaced}}:
> after the service is created the OM returns a new metadata manager, and one
> run must clean the expired entry of the new table and not the one of the
> original table. Without the change it submits the original table's entry.
> With it {{TestRevokedSTSTokenCleanupService}},
> {{TestS3RevokeSTSTokenRequest}} and {{TestS3SecurityUtil}} pass (38 tests)
> and checkstyle is clean.
> Found by code review of the OM Ratis snapshot (checkpoint) install paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
