Wei-Chiu Chuang created HDDS-10300:
--------------------------------------
Summary: InstallSnapshot may fail if OM metadata dir and OM DB dir
are in different local storage partitions
Key: HDDS-10300
URL: https://issues.apache.org/jira/browse/HDDS-10300
Project: Apache Ozone
Issue Type: Bug
Reporter: Wei-Chiu Chuang
OM DB snapshot installation requires hard link, which can fail if OM metadata
dir and OM DB dir are in different partitions.
{noformat}
2024-01-31 14:14:59,406 ERROR
[pool-33-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to move
downloaded DB checkpoint
/dfs1/hadoop-ozone/om/ozone-metadata/snapshot/om.db.candidate to metadata
directory /ssd/hadoop-ozone/om/data/om.db. Exception: {}. Resetting to original
DB.
java.nio.file.FileSystemException: /ssd/hadoop-ozone/om/data/om.db/000031.sst
-> /dfs1/hadoop-ozone/om/ozone-metadata/snapshot/om.db.candidate/000031.sst:
Invalid cross-device link
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
at java.nio.file.Files.createLink(Files.java:1086)
at
org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
at
org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3816)
at
org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3796)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3671)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3606)
at
org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3583)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$6(OzoneManagerStateMachine.java:513)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
2024-01-31 14:14:59,406 ERROR
[pool-33-thread-1]-org.apache.hadoop.ozone.om.OzoneManager: Failed to install
Snapshot from om3 as OM failed to replace DB with downloaded checkpoint.
Reloading old OM state.
java.nio.file.FileSystemException: /ssd/hadoop-ozone/om/data/om.db/000031.sst
-> /dfs1/hadoop-ozone/om/ozone-metadata/snapshot/om.db.candidate/000031.sst:
Invalid cross-device link
at
sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at
sun.nio.fs.UnixFileSystemProvider.createLink(UnixFileSystemProvider.java:476)
at java.nio.file.Files.createLink(Files.java:1086)
at
org.apache.hadoop.ozone.om.snapshot.OmSnapshotUtils.linkFiles(OmSnapshotUtils.java:169)
at
org.apache.hadoop.ozone.om.OzoneManager.moveCheckpointFiles(OzoneManager.java:3816)
at
org.apache.hadoop.ozone.om.OzoneManager.replaceOMDBWithCheckpoint(OzoneManager.java:3796)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3671)
at
org.apache.hadoop.ozone.om.OzoneManager.installCheckpoint(OzoneManager.java:3606)
at
org.apache.hadoop.ozone.om.OzoneManager.installSnapshotFromLeader(OzoneManager.java:3583)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$6(OzoneManagerStateMachine.java:513)
at
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
