[
https://issues.apache.org/jira/browse/HDDS-12481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17936847#comment-17936847
]
Swaminathan Balachandran commented on HDDS-12481:
-------------------------------------------------
[~jyosin] Can you please attach om logs for this? The logs here are just not
sufficient. We need to see what led to the rocksdb close.
> OM down due to OMDoubleBuffer error
> -----------------------------------
>
> Key: HDDS-12481
> URL: https://issues.apache.org/jira/browse/HDDS-12481
> Project: Apache Ozone
> Issue Type: Bug
> Components: Snapshot
> Reporter: Jyotirmoy Sinha
> Priority: Major
>
> Scenario -
> # 150 Volumes
> # 4 buckets per volume of combinations -
> ## ratis - fso
> ## ratis - obs
> ## ec - fso
> ## ec -obs
> # 7 snapshots -
> ## 4 backup snapshots
> ## 3 replication snapshots
> # Total number of snapshots = 150 * 4 * 7 = 4200
> # 500K keys per snapshot (key scale to be configurable)
> # Repeat steps 3-6 for continuous intervals
> OM Error stacktrace -
> {code:java}
> 2025-03-03 09:53:30,921 INFO
> [grpc-default-executor-644]-org.apache.ratis.grpc.server.GrpcLogAppender:
> om122@group-9E840F57059A->om121-InstallSnapshotResponseHandler:
> InstallSnapshot in progress.
> 2025-03-03 09:53:30,921 ERROR
> [OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer:
> Terminating with exit status 1: During flush to DB encountered error in
> OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling
> OMRequest: cmdType: CreateSnapshot
> traceID: ""
> success: true
> status: OK
> CreateSnapshotResponse {
> snapshotInfo {
> snapshotID {
> mostSigBits: 7137775835294879303
> leastSigBits: -8902643940451942791
> }
> name: "snap1741024398"
> volumeName: "bofavol-38"
> bucketName: "bofabuck-38-ratfso"
> snapshotStatus: SNAPSHOT_ACTIVE
> creationTime: 1741024409564
> deletionTime: 18446744073709551615
> pathPreviousSnapshotID {
> mostSigBits: -1057436070549500687
> leastSigBits: -5683722903965903247
> }
> globalPreviousSnapshotID {
> mostSigBits: 8988231063147725793
> leastSigBits: -5103262545165094617
> }
> snapshotPath: "bofavol-38/bofabuck-38-ratfso"
> checkpointDir: "-630e794d-fd1e-4e47-8473-74811f93fe79"
> dbTxSequenceNumber: 323480877
> deepClean: true
> sstFiltered: false
> }
> } java.io.IOException: Rocks Database is closed
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:439)
> at
> org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:777)
> at
> org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:232)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:418)
> at
> org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:55)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:558)
> at
> org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:437)
> at
> org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
> at
> org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:383)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:221)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:382)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:325)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:298)
> at
> org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:263)
> at java.lang.Thread.run(Thread.java:748){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
