[
https://issues.apache.org/jira/browse/HDDS-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17291:
------------------------------
Attachment: TestBugMC1PrunerSnapshotDiffCheckpointDeadlock.java
> SST value pruner waits for the bootstrap lock while holding the differ
> monitor, so OM bootstrap can deadlock with snapshot diff or defrag
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17291
> URL: https://issues.apache.org/jira/browse/HDDS-17291
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-take-bootstrap-lock-before-differ-monitor-in-sst-pruner.patch,
> TestBugMC1PrunerSnapshotDiffCheckpointDeadlock.java,
> TestBugMC2DefragPrunerCheckpointDeadlock.java
>
>
> h3. Mechanism
> {{RocksDBCheckpointDiffer.pruneSstFileValues}} enters {{synchronized (this)}}
> for each compaction log entry, rewrites the entry's backup SST files and,
> still inside the monitor, takes the BOOTSTRAP read lock (key
> {{rocksdbCheckpointDifferAOS}}, supplied by {{OmMetadataManagerImpl.loadDB}})
> around the {{Files.move}} that replaces each file. Apart from
> {{loadAllCompactionLogs}}, which runs once while the DB is opened and removes
> orphaned backup files under the bootstrap read lock inside the monitor, every
> other user of the two locks takes them in the opposite order:
> {{pruneOlderSnapshotsWithCompactionHistory}} and {{SnapshotDefragService}}
> take the bootstrap lock first, and a snapshot diff job holds SNAPSHOT_DB read
> locks on both snapshots ({{SnapshotCache.get}}) when
> {{RDBDifferComputer.computeDeltaFiles}} enters {{synchronized (differ)}}. The
> differ monitor is not a tracked lock resource, so the DAG lock tracker does
> not catch this.
> The checkpoint servlet ({{OMDBCheckpointServletInodeBasedXfer}}) write-locks
> every BOOTSTRAP stripe through {{OMDBCheckpointServlet.Lock}} and holds them
> while it collects the files. A follower bootstrap always requests snapshot
> data, so in its last batch the servlet also write-locks every SNAPSHOT_DB
> stripe through {{SnapshotCache.lock()}}. That gives two cycles:
> * *Snapshot diff (default configuration).* The pruner holds the monitor and
> waits for BOOTSTRAP read, which the servlet holds. The snapshot diff job
> holds SNAPSHOT_DB read and waits for the monitor. The servlet holds BOOTSTRAP
> write and waits for SNAPSHOT_DB write.
> * *Defrag (non default stripe count).* {{SnapshotDefragService}} holds
> BOOTSTRAP read with key {{SnapshotDefragService}} and waits for the monitor
> in {{RDBDifferComputer.computeDeltaFiles}}. The pruner holds the monitor and
> waits for BOOTSTRAP read on its own stripe. The servlet, which takes the
> stripes in ascending order, has already taken the pruner's stripe (or is
> queued on it) and waits for the defrag stripe. This needs the pruner's stripe
> at or below the defrag stripe. Both keys are constants, so the outcome is
> fixed by {{ozone.om.lock.stripes.bootstrap_lock}}: no cycle at the default
> 512 (or 32, 1024), a cycle at 1, 2, 4, 8, 16, 64, 128, 256 and 2048 to 32768
> (computed from the Guava stripe mapping; other values round up to the next
> power of two, so 100 behaves like 128). That key is not documented.
> h3. Trigger
> Both cycles need the rocks-tools native library, which Linux release builds
> include (built with {{-Drocks_tools_native}}) and the OM loads by default;
> without it the pruner is not scheduled. The 2.2.1 library does not load on
> RHEL or Rocky Linux 9, including the {{apache/ozone:2.2.1}} image
> (HDDS-17153, fixed in 2.3.0), so neither cycle forms on those hosts.
> Snapshots must exist and a tracked column family must have been compacted
> since, so that the prune queue is not empty.
> For the snapshot diff cycle a follower must be bootstrapping from the leader,
> because only a follower bootstrap requests snapshot data and so makes the
> servlet take {{SnapshotCache.lock()}}. The pruner's periodic run (every 10
> minutes by default) has to start while the servlet holds the bootstrap lock,
> or be in its rewrite when the servlet arrives. Either way the pruner ends up
> parked on the bootstrap lock while holding the monitor. A snapshot diff job
> that reaches its delta file computation before the servlet reaches
> {{SnapshotCache.lock()}} then closes the cycle. The servlet holds the
> bootstrap lock for its whole collection phase, so this window is as long as
> copying the snapshot directories (argued from the code; the reproduction
> orders the steps with a latch).
> For the defrag cycle, {{ozone.snapshot.defrag.service.interval}} must be
> enabled (it is off by default) and the stripe count must be one of the values
> above. Any request that takes the BOOTSTRAP write lock is enough, not only a
> follower bootstrap: Recon's OM DB fetch (which does not request snapshot
> data) and the v1 {{/dbCheckpoint}} servlet take the same lock.
> h3. Impact
> All waits are untimed lock or monitor waits, and the JVM's deadlock detection
> does not report them because read locks have no owner. Only an OM restart
> recovers. On the leader:
> * the checkpoint request never returns, so the follower cannot finish its
> bootstrap; a retry queues behind the same lock;
> * every later user of the BOOTSTRAP read lock blocks behind the servlet: the
> key, directory and snapshot deleting services when they submit purges,
> {{SstFilteringService}}, {{SnapshotDefragService}} and the differ's pruning
> tasks, so space is no longer reclaimed;
> * later snapshot diff jobs block on the differ monitor. Snapshot reads and
> diffs whose snapshots map to the SNAPSHOT_DB stripes the servlet has already
> taken also block (argued, not run).
> In the defrag cycle the servlet waits at the defrag stripe, so only bootstrap
> readers on that stripe and the stripes below it block, and snapshot reads are
> not affected. The requester (follower or Recon) never gets its checkpoint,
> and defrag, the differ's pruning tasks (they share the pruner's thread) and
> later snapshot diff jobs hang as above.
> Released versions 2.2.0 and 2.2.1 have the same code (read from the tags).
> 2.1.x has the nesting, but its servlet does not take
> {{SnapshotCache.lock()}}, so the snapshot diff cycle does not form there.
> h3. Reproduction
> Two self contained classes for {{hadoop-ozone/ozone-manager}} (target path
> and Maven command in the class comment). Both pass on unmodified source when
> the bug is present. They use the real {{OzoneManagerLock}}, the OM DB and
> differ built by {{OmMetadataManagerImpl.loadDB}} (so the production lock
> supplier), a prune queue entry produced by a real compaction of keyTable with
> a snapshotInfoTable row present, {{pruneSstFileValues}},
> {{RDBDifferComputer.computeDeltaFiles}} and the servlet's
> {{OMDBCheckpointServlet.Lock}}. The native SST reader is mocked as in
> {{TestRocksDBCheckpointDiffer}}, so they run without the native library, and
> the snapshot local data provider is mocked. The servlet request, the snapshot
> diff job and the defrag task are replayed at their lock steps, not run end to
> end through HTTP or the services.
> * [^TestBugMC1PrunerSnapshotDiffCheckpointDeadlock.java]: default
> configuration. A latch in the mocked SST scan keeps the pruner in its rewrite
> until the snapshot diff thread is blocked on the monitor and the servlet
> thread waits in {{SnapshotCache.lock()}}. After the latch opens none of the
> three finishes; the pruner waits on the lock owned by the servlet thread, the
> snapshot diff thread on the monitor owned by the pruner, and a later
> bootstrap read (key {{KeyDeletingService}}) blocks too.
> * [^TestBugMC2DefragPrunerCheckpointDeadlock.java]: defrag cycle with the
> bootstrap lock of a real {{SnapshotDefragService}}, parameterized over 2, 256
> and 2048 stripes (all three threads stuck, defrag blocked on the monitor
> owned by the pruner) and a control at the default 512 (all three complete).
> Not exercised: a real follower bootstrap over HTTP, a real snapshot diff or
> defrag run, and the real native library.
> h3. Patch
> [^MC-1-take-bootstrap-lock-before-differ-monitor-in-sst-pruner.patch],
> against e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master
> at 1ff29ee6b86.
> {{pruneSstFileValues}} now takes the BOOTSTRAP read lock for each compaction
> log entry before entering {{synchronized (this)}}, instead of around
> {{Files.move}} inside it. The pruner then never waits for the bootstrap lock
> while holding the monitor, which removes the edge both cycles need, at any
> stripe count. The move is still done under the bootstrap read lock, so a
> checkpoint that holds the write lock still cannot see a backup SST being
> replaced (the protection added by HDDS-12501 and HDDS-13905). The rest of the
> method is unchanged apart from indentation ({{git diff -w}} shows only the
> real change).
> The cost is a longer wait for a bootstrap. The lock now covers one compaction
> log entry, whose input file list can be long after a large or manual
> compaction, and the pruner holds it while it waits for the differ monitor, so
> a bootstrap that arrives then also waits for any snapshot diff or defrag
> already in the monitor. While the servlet waits on the pruner's stripe it
> holds the stripes below it, so bootstrap readers on those stripes wait as
> well. The servlet is not starved: the pruner releases the lock between
> entries, and with the default nonfair lock its next read waits behind the
> queued writer.
> After the patch {{loadAllCompactionLogs}} is the only path left that takes
> the monitor and then the bootstrap lock (through
> {{cleanupOrphanedSstBackupFiles}}, added by HDDS-14742 after 2.2.1). It
> cannot deadlock: it runs once in the {{RDBStore}} constructor, before the
> pruner's first run (scheduled one interval after start) and before this OM
> serves checkpoint requests. The patch leaves it unchanged.
> New test
> {{TestRocksDBCheckpointDiffer#testPruneSstFileValuesWaitsForBootstrapLockOutsideDifferMonitor}}:
> while another thread holds the bootstrap write lock it checks that the
> backup SST file is unchanged and that a snapshot diff style {{synchronized
> (differ)}} can proceed, then that the file is pruned once the lock is
> released. It fails without the change. With it the
> {{rocksdb-checkpoint-differ}} module tests (121), the related
> {{ozone-manager}} unit tests (141, including {{TestSnapshotDefragService}},
> {{TestSnapshotDiffManager}}, {{TestRDBDifferComputer}}) and
> {{TestOMDbCheckpointServletInodeBasedXfer}} (15) pass, and both attached
> reproductions no longer show the deadlock.
> Found by TLA+ model checking and code review of the Ozone Manager lock
> ordering paths under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-13905 (resolved, made background bootstrap locks read locks
> and kept this nesting), HDDS-9486 (resolved, an earlier pruner and checkpoint
> deadlock), HDDS-16685 (snapshot diff takes SNAPSHOT_DB stripes out of order
> against the same servlet step) and HDDS-17259 (the compaction listener
> synchronizes on the wrong object in the same class). The attached patch is a
> proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
