Li Cheng created HDDS-2396:
------------------------------
Summary: OM rocksdb core dump during writing
Key: HDDS-2396
URL: https://issues.apache.org/jira/browse/HDDS-2396
Project: Hadoop Distributed Data Store
Issue Type: Bug
Components: Ozone Manager
Affects Versions: 0.4.1
Reporter: Li Cheng
Attachments: hs_err_pid9340.log
Env: 4 VMs in total: 3 Datanodes on 3 VMs, 1 OM & 1 SCM on a separate VM, say
it's VM0.
I use goofys as a fuse and enable ozone S3 gateway to mount ozone to a path on
VM0, while reading data from VM0 local disk and write to mount path. The
dataset has various sizes of files from 0 byte to GB-level and it has a number
of ~50,000 files.
There happens core dump in rocksdb while it's occasional.
Stack: [0x00007f5891a23000,0x00007f5891b24000], sp=0x00007f5891b21bb8, free
space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x151d60] __memmove_ssse3_back+0x1ae0
C [librocksdbjni3192271038586903156.so+0x358fec]
rocksdb::MemTableInserter::PutCFImpl(unsigned int, rocksdb::Slice const&,
rocksdb::Slice const&, rocksdb:
:ValueType)+0x51c
C [librocksdbjni3192271038586903156.so+0x359d17]
rocksdb::MemTableInserter::PutCF(unsigned int, rocksdb::Slice const&,
rocksdb::Slice const&)+0x17
C [librocksdbjni3192271038586903156.so+0x3513bc]
rocksdb::WriteBatch::Iterate(rocksdb::WriteBatch::Handler*) const+0x45c
C [librocksdbjni3192271038586903156.so+0x354df9]
rocksdb::WriteBatchInternal::InsertInto(rocksdb::WriteThread::WriteGroup&,
unsigned long, rocksdb::ColumnFamilyMemTables*, rocksdb::FlushScheduler*, bool,
unsigned long, rocksdb::DB*, bool, bool, bool)+0x1f9
C [librocksdbjni3192271038586903156.so+0x29fd79]
rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*,
rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*,
unsigned long, rocksdb::PreReleaseCallback*)+0x24b9
C [librocksdbjni3192271038586903156.so+0x2a0431]
rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x21
C [librocksdbjni3192271038586903156.so+0x1a064c]
Java_org_rocksdb_RocksDB_write0+0xcc
J 7899 org.rocksdb.RocksDB.write0(JJJ)V (0 bytes) @ 0x00007f58f1872dbe
[0x00007f58f1872d00+0xbe]
J 10093% C1
org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions()V
(400 bytes) @ 0x00007f58f2308b0c [0x00007f58f2307a40+0x10cc]
j org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer$$Lambda$29.run()V+4
j java.lang.Thread.run()V+11
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
