Siyao Meng created HDDS-13062:
---------------------------------
Summary: Verify that ozone.metadata.dirs and ozone.om.db.dirs to
be in the same mount upon OM startup
Key: HDDS-13062
URL: https://issues.apache.org/jira/browse/HDDS-13062
Project: Apache Ozone
Issue Type: Improvement
Reporter: Siyao Meng
When {{ozone.metadata.dirs}} and {{ozone.om.db.dirs}} are set to different
mounts, the following exception would occur:
{code}
2025-05-03 20:49:12,016 ERROR
[pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to move
downloaded DB checkpoint
/hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate to metadata
directory /hadoop-ozone/datanode02/om/data/om.db. Exception: {}. Resetting to
original DB.
java.nio.file.FileSystemException:
/hadoop-ozone/datanode02/om/data/om.db/000079.sst ->
/hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst:
Invalid cross-device link
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
at java.nio.file.Files.createLink(Files.java:1086)
at
org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
at
org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
at
org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
at
org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2025-05-03 20:49:12,016 ERROR
[pool-30-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to install
Snapshot from om1546339340 as OM failed to replace DB with downloaded
checkpoint. Reloading old OM state.
java.nio.file.FileSystemException:
/hadoop-ozone/datanode02/om/data/om.db/000079.sst ->
/hadoop-ozone/datanode01/om/ozone-metadata/snapshot/om.db.candidate/000079.sst:
Invalid cross-device link
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
at java.nio.file.Files.createLink(Files.java:1086)
at
org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
at
org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3880)
at
org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3860)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3734)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3669)
at
org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3646)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$5(OzoneManagerStateMachine.java:505)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{code}
At this point, the follower OM won't ever succeed bootstrapping and will lag
behind the leader indefinitely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
