Siyao Meng created HDDS-17461:
---------------------------------
Summary: After a checkpoint install, OM volume, bucket and
snapshot directory metrics keep reading the replaced metadata manager
Key: HDDS-17461
URL: https://issues.apache.org/jira/browse/HDDS-17461
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-9a-recreate-metadata-metrics-after-checkpoint-install.patch,
TestBugCR9StateReloadStaleMetricsAndCacheThreads.java
h3. Mechanism
{{VolumeUtilizationMetrics}} and {{BucketUtilizationMetrics}} are created once
in the {{OzoneManager}} constructor and keep a final reference to the startup
{{OmMetadataManagerImpl}}; every collection iterates that manager's volume or
bucket cache. {{OMSnapshotDirectoryMetrics}} likewise keeps the manager it was
created with.
{{OzoneManager.reloadOMState}}, run by {{installCheckpoint}}, builds a new
metadata manager and re-wires the managers and services that hold it, but not
these three sources. So after an install they keep reading the stopped manager:
the utilization metrics report the volumes and buckets as they were before the
install, and the snapshot directory metrics report zeros because the old
manager's store is closed.
{{OMMetrics.startSnapshotDirectoryMetrics}} only creates the source when none
exists, so the stop and start pair in {{OzoneManager.restart}} (used by the
mini cluster) does not switch it to the new manager either, and {{restart}}
never registers the two utilization sources again after {{stop}} unregistered
them.
h3. Trigger
A follower OM installs a checkpoint from the leader, which is the normal catch
up for an OM that was down while the leader purged the log it needs (or a newly
bootstrapped OM). No fault or operator action is needed.
h3. Impact
On that OM, the per volume and per bucket utilization metrics and the snapshot
directory metrics are wrong until the OM process restarts: volumes and buckets
created after the install are missing, deleted ones remain, and usage values
are frozen. Requests and data are not affected. Volume utilization metrics
exist only on master; the bucket metrics are released since 2.0.0 and the
snapshot directory metrics since 2.2.0.
h3. Reproduction
[^TestBugCR9StateReloadStaleMetricsAndCacheThreads.java]
({{hadoop-ozone/integration-test}}, target path and Maven command in the class
comment) runs three OMs without datanodes with one OM not started, creates 120
volumes and a bucket past a purge gap of 50, then starts that OM and waits
until Ratis has installed a checkpoint on it and it caught up with the leader.
Its DB then has every volume and the bucket, but its
{{VolumeUtilizationMetrics}} report only {{s3v}}, its
{{BucketUtilizationMetrics}} report no bucket, and its snapshot directory
metrics do not read its current metadata manager. The test also checks that the
replaced table caches keep their cleanup executors running, which HDDS-17460
covers. A passing test means the defects are present; with this patch the three
metrics checks fail.
h3. Patch
[^CR-9a-recreate-metadata-metrics-after-checkpoint-install.patch], against
a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
b6d0181075e.
A new private {{OzoneManager.restartMetadataManagerMetrics}}, called from both
{{reloadOMState}} and {{restart}}, unregisters the two utilization sources and
creates them again for the current metadata manager, and stops and starts the
snapshot directory metrics (replacing the pair {{restart}} already had).
{{OMMetrics.stopSnapshotDirectoryMetrics}} now unregisters the source and drops
it, so the next start creates one for the manager it is given. This is the
pattern HDDS-17038 used for the double buffer metrics on the same install path.
Every source is unregistered before it is registered again; unregistering a
source that {{stop}} already removed does nothing.
Covered by new checks at the end of {{testInstallSnapshotWithClientWrite}} in
the existing {{TestOMRatisSnapshots}}: after the install, the follower's volume
and bucket utilization sources must report the test volume and bucket, and its
snapshot directory metrics must read its current metadata manager. Without the
change the volume check fails ({{["s3v"]}} does not contain the test volume),
and without the {{OMMetrics}} part the snapshot directory check fails. A new
check in {{testRestartOMWithVolumeOperation}} of the existing
{{TestOzoneManagerRestart}} asserts that the volume utilization source is
registered after {{restart}}; it fails without the change. With the change
{{TestOMRatisSnapshots}}, {{TestOMInstallSnapshotDuringBootstrapping}},
{{TestOzoneManagerRestart}}, {{TestOzoneManagerHAWithStoppedNodes}} (which
restarts OMs), {{TestOmMetrics}} and the unit tests of the three metrics
classes and {{OMMetrics}} pass, and checkstyle is clean.
{{TestOMRatisSnapshotTransfer}} timed out here downloading the checkpoint, with
and without the patch, before any reload, so it was not seen passing.
Found by code review of the OM table cache, double buffer flush and startup
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing (nearest: HDDS-17038,
the same defect for the double buffer metrics, fixed with the same pattern;
HDDS-10476, HDDS-14041 and HDDS-15333, which added these metrics). The attached
patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
