jojochuang commented on code in PR #11009:
URL: https://github.com/apache/ozone/pull/11009#discussion_r3859945675
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java:
##########
@@ -658,43 +670,94 @@ void processDeletedDirsForStore(SnapshotInfo
currentSnapshotInfo, KeyManager key
UUID expectedPreviousSnapshotId = currentSnapshotInfo == null ?
snapshotChainManager.getLatestGlobalSnapshotId() :
SnapshotUtils.getPreviousSnapshotId(currentSnapshotInfo,
snapshotChainManager);
- Map<UUID, Pair<Long, Long>> exclusiveSizeMap = Maps.newConcurrentMap();
-
- CompletableFuture<Boolean> processedAllDeletedDirs =
CompletableFuture.completedFuture(true);
final int parallelThreads = numberOfParallelThreadsPerStore.get();
+ CountDownLatch snapshotDbHandlesClosed = currentSnapshotInfo == null ?
null :
Review Comment:
**Possible remaining deadlock (AOS, cross-worker)**
The snapshot barrier is skipped when `currentSnapshotInfo == null`, but AOS
still fans out `parallelThreads` workers (default 10). Each worker calls
`closeSnapshotDbHandles()` in `beforeSubmit` before **its own** submit, yet
there is no cross-worker gate like `snapshotDbHandlesClosed`.
So worker A can be blocked in synchronous `submitRequest(PurgeDirectories)`
waiting for double-buffer capacity while worker B is still scanning with
previous-snapshot `SNAPSHOT_DB` read locks open via
`ReclaimableFilter.apply()`. That is the same circular wait class as HDDS-16164:
```
Worker A: closed own handles → submitRequest → waits for unflushed slot
Worker B: still scanning → holds SNAPSHOT_DB read lock
Purge flush: holds slot → needs SNAPSHOT_DB write on colliding stripe →
waits on B
```
`KeyDeletingService` avoids this by closing all handles once before any
submit on a single worker thread. Consider extending the barrier to AOS, or
serializing AOS submit after all workers finish scanning (even if scan stays
parallel). With `OZONE_THREAD_NUMBER_DIR_DELETION=1` this particular
cross-worker variant goes away.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/DirectoryDeletingService.java:
##########
@@ -824,12 +920,25 @@ public BackgroundTaskResult call() {
} else if (!isPreviousPurgeTransactionFlushed()) {
return BackgroundTaskResult.EmptyTaskResult.newResult();
}
- try (UncheckedAutoCloseableSupplier<OmSnapshot> omSnapshot =
snapInfo == null ? null :
- omSnapshotManager.getActiveSnapshot(snapInfo.getVolumeName(),
snapInfo.getBucketName(),
- snapInfo.getName())) {
- KeyManager keyManager = snapInfo == null ?
getOzoneManager().getKeyManager()
- : omSnapshot.get().getKeyManager();
- processDeletedDirsForStore(snapInfo, keyManager, run,
pathLimitPerTask);
+ boolean snapshotProcessingLockAcquired = false;
Review Comment:
**Possible remaining deadlock (AOS task concurrent with snapshot purge)**
`snapshotProcessingLock` serializes snapshot-vs-snapshot tasks, but AOS
(`snapshotId == null`) never takes it. `BackgroundService` still runs the AOS
`DirDeletingTask` concurrently with snapshot deep-clean / purge work.
Even with the snapshot path fully fixed, an AOS worker can retain
previous-snapshot `SNAPSHOT_DB` read locks while another thread's snapshot
purge occupies the double buffer and its flush needs a colliding write lock:
```
AOS worker B: scanning → SNAPSHOT_DB read on snapshot S
Snapshot purge for S: holds unflushed slot → flush waits for write lock on S
AOS worker A: finished scan, closed handles → submitRequest blocks on slot
```
Worth tracking as a follow-up if we want DDS-wide protection, not just
snapshot deep-clean. Options: coarse coordination between AOS and
snapshot/purge paths, or reuse the same "all workers released DB handles before
any submit" invariant for AOS.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
