I think you hit the os process fd limit. You need to adjust it. -Sam
On Wed, Jun 15, 2016 at 2:07 PM, Mansour Shafaei Moghaddam <mansoor.shaf...@gmail.com> wrote: > It fails at "FileStore.cc: 2761". Here is a more complete log: > > -9> 2016-06-15 10:55:13.205014 7fa2dcd85700 -1 dump_open_fds unable to > open /proc/self/fd > -8> 2016-06-15 10:55:13.205085 7fa2cb402700 2 > filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > > 104857600 > -7> 2016-06-15 10:55:13.205094 7fa2cd406700 2 > filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328389 > > 104857600 > -6> 2016-06-15 10:55:13.205111 7fa2cac01700 2 > filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328317 > > 104857600 > -5> 2016-06-15 10:55:13.205118 7fa2ca400700 2 > filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > > 104857600 > -4> 2016-06-15 10:55:13.205121 7fa2cdc07700 2 > filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 > > 104857600 > -3> 2016-06-15 10:55:13.205153 7fa2de588700 5 -- op tracker -- seq: > 1476, time: 2016-06-15 10:55:13.205153, event: journaled_completion_queued, > op: osd_op(client.4109.0:1457 rb.0.100a.6b8b4567.000000006b6c > [set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~4096] > 0.cbe1d8a4 ack+ondisk+write e9) > -2> 2016-06-15 10:55:13.205183 7fa2de588700 5 -- op tracker -- seq: > 1483, time: 2016-06-15 10:55:13.205183, event: > write_thread_in_journal_buffer, op: osd_op(client.4109.0:1464 > rb.0.100a.6b8b4567.00000000524d [set-alloc-hint object_size 4194304 > write_size 4194304,write 3051520~4096] 0.6778c255 ack+ondisk+write e9) > -1> 2016-06-15 10:55:13.205400 7fa2de588700 5 -- op tracker -- seq: > 1483, time: 2016-06-15 10:55:13.205400, event: journaled_completion_queued, > op: osd_op(client.4109.0:1464 rb.0.100a.6b8b4567.00000000524d > [set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~4096] > 0.6778c255 ack+ondisk+write e9) > 0> 2016-06-15 10:55:13.206559 7fa2dcd85700 -1 os/FileStore.cc: In > function 'unsigned int FileStore::_do_transaction(ObjectStore::Transaction&, > uint64_t, int, ThreadPool::TPHandle*)' thread 7fa2dcd85700 time 2016-06-15 > 10:55:13.205018 > os/FileStore.cc: 2761: FAILED assert(0 == "unexpected error") > > ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x78) [0xacd718] > 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, > int, ThreadPool::TPHandle*)+0xa24) [0x8b8114] > 3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, > std::allocator<ObjectStore::Transaction*> >&, unsigned long, > ThreadPool::TPHandle*)+0x64) [0x8bcf34] > 4: (FileStore::_do_op(FileStore::OpSequencer*, > ThreadPool::TPHandle&)+0x17e) [0x8bd0ce] > 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326] > 6: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0] > 7: (()+0x7dc5) [0x7fa2e88f3dc5] > 8: (clone()+0x6d) [0x7fa2e73d528d] > > > On Wed, Jun 15, 2016 at 2:05 PM, Somnath Roy <somnath....@sandisk.com> > wrote: >> >> There should be a line in the log specifying which assert is failing , >> post that along with say 10 lines from top of that.. >> >> >> >> Thanks & Regards >> >> Somnath >> >> >> >> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of >> Mansour Shafaei Moghaddam >> Sent: Wednesday, June 15, 2016 1:57 PM >> To: ceph-users@lists.ceph.com >> Subject: [ceph-users] Fio randwrite does not work on Centos 7.2 VM >> >> >> >> Hi All, >> >> >> >> Has anyone faced a similar issue? I do not have a problem with random >> read, sequential read, and sequential writes though. Everytime I try running >> fio for random writes, one osd in the cluster crashes. Here is the what I >> see at the tail of the log: >> >> >> >> ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403) >> >> 1: ceph-osd() [0x9d6334] >> >> 2: (()+0xf100) [0x7fa2e88fb100] >> >> 3: (gsignal()+0x37) [0x7fa2e73145f7] >> >> 4: (abort()+0x148) [0x7fa2e7315ce8] >> >> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fa2e7c189d5] >> >> 6: (()+0x5e946) [0x7fa2e7c16946] >> >> 7: (()+0x5e973) [0x7fa2e7c16973] >> >> 8: (()+0x5eb93) [0x7fa2e7c16b93] >> >> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x24a) [0xacd8ea] >> >> 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, >> int, ThreadPool::TPHandle*)+0xa24) [0x8b8114] >> >> 11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*, >> std::allocator<ObjectStore::Transaction*> >&, unsigned long, >> ThreadPool::TPHandle*)+0x64) [0x8bcf34] >> >> 12: (FileStore::_do_op(FileStore::OpSequencer*, >> ThreadPool::TPHandle&)+0x17e) [0x8bd0ce] >> >> 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326] >> >> 14: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0] >> >> 15: (()+0x7dc5) [0x7fa2e88f3dc5] >> >> 16: (clone()+0x6d) [0x7fa2e73d528d] >> >> >> >> >> >> PLEASE NOTE: The information contained in this electronic mail message is >> intended only for the use of the designated recipient(s) named above. If the >> reader of this message is not the intended recipient, you are hereby >> notified that you have received this message in error and that any review, >> dissemination, distribution, or copying of this message is strictly >> prohibited. If you have received this communication in error, please notify >> the sender by telephone or e-mail (as shown above) immediately and destroy >> any and all copies of this message in your possession (whether hard copies >> or electronically stored copies). > > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
