smengcl commented on PR #9445: URL: https://github.com/apache/ozone/pull/9445#issuecomment-6515830283
fyi, for anyone backporting this PR to a branch without the later STS branch merge: this commit creates `s3RevokedStsTokenTable` with a hardcoded `CacheType.FULL_CACHE` in `OmMetadataManagerImpl#initializeOmTables`, instead of the `cacheType` argument. `initializeOmTables` also builds snapshot and checkpoint DBs, which pass `PARTIAL_CACHE` (HDDS-9778). This table ignores that, so every snapshot DB open creates a `FullTableCache`. Its scheduled cleanup thread starts in the constructor and is never shut down, so each open leaks one `s3RevokedStsTokenTable_FullTableCache-Cleanup-0` thread. Snapshot DBs are reopened constantly (snapshot cache, KeyDeletingService, SnapshotDeletingService, snapdiff, defrag). With this commit backported, a sustained snapshot create, delete and snapdiff test reached about 170K of these threads per OM at about 18K snapshots. Each thread wakes once a second, so G1 young pauses grew from about 8 ms to about 300 ms, and OM leadership kept moving between the OMs until clients could not find a leader. On master this is already fixed: the STS branch merge f7c62ae232933cebd0d4deca3de1fd18f1fd369d changed the line to `cacheType`. That change only exists in the merge commit, so cherry-picking the STS commits alone leaves the bug in. Please pick up that one line together with this commit. The generic gap that table cache executors are never shut down is tracked in HDDS-17460. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
