It fails at "FileStore.cc: 2761". Here is a more complete log:
-9> 2016-06-15 10:55:13.205014 7fa2dcd85700 -1 dump_open_fds unable to
open /proc/self/fd
-8> 2016-06-15 10:55:13.205085 7fa2cb402700 2
filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 >
104857600
-7> 2016-06-15 10:55:13.205094 7fa2cd406700 2
filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328389 >
104857600
-6> 2016-06-15 10:55:13.205111 7fa2cac01700 2
filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328317 >
104857600
-5> 2016-06-15 10:55:13.205118 7fa2ca400700 2
filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 >
104857600
-4> 2016-06-15 10:55:13.205121 7fa2cdc07700 2
filestore(/var/lib/ceph/osd/ceph-0) waiting 51 > 50 ops || 328390 >
104857600
-3> 2016-06-15 10:55:13.205153 7fa2de588700 5 -- op tracker -- seq:
1476, time: 2016-06-15 10:55:13.205153, event: journaled_completion_queued,
op: osd_op(client.4109.0:1457 rb.0.100a.6b8b4567.000000006b6c
[set-alloc-hint object_size 4194304 write_size 4194304,write 1884160~4096]
0.cbe1d8a4 ack+ondisk+write e9)
-2> 2016-06-15 10:55:13.205183 7fa2de588700 5 -- op tracker -- seq:
1483, time: 2016-06-15 10:55:13.205183, event:
write_thread_in_journal_buffer, op: osd_op(client.4109.0:1464
rb.0.100a.6b8b4567.00000000524d [set-alloc-hint object_size 4194304
write_size 4194304,write 3051520~4096] 0.6778c255 ack+ondisk+write e9)
-1> 2016-06-15 10:55:13.205400 7fa2de588700 5 -- op tracker -- seq:
1483, time: 2016-06-15 10:55:13.205400, event: journaled_completion_queued,
op: osd_op(client.4109.0:1464 rb.0.100a.6b8b4567.00000000524d
[set-alloc-hint object_size 4194304 write_size 4194304,write 3051520~4096]
0.6778c255 ack+ondisk+write e9)
0> 2016-06-15 10:55:13.206559 7fa2dcd85700 -1 os/FileStore.cc: In
function 'unsigned int
FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t, int,
ThreadPool::TPHandle*)' thread 7fa2dcd85700 time 2016-06-15 10:55:13.205018
os/FileStore.cc: 2761: FAILED assert(0 == "unexpected error")
ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x78) [0xacd718]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
int, ThreadPool::TPHandle*)+0xa24) [0x8b8114]
3: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*,
std::allocator<ObjectStore::Transaction*> >&, unsigned long,
ThreadPool::TPHandle*)+0x64) [0x8bcf34]
4: (FileStore::_do_op(FileStore::OpSequencer*,
ThreadPool::TPHandle&)+0x17e) [0x8bd0ce]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326]
6: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0]
7: (()+0x7dc5) [0x7fa2e88f3dc5]
8: (clone()+0x6d) [0x7fa2e73d528d]
On Wed, Jun 15, 2016 at 2:05 PM, Somnath Roy <somnath....@sandisk.com>
wrote:
> There should be a line in the log specifying which assert is failing ,
> post that along with say 10 lines from top of that..
>
>
>
> Thanks & Regards
>
> Somnath
>
>
>
> *From:* ceph-users [mailto:ceph-users-boun...@lists.ceph.com] *On Behalf
> Of *Mansour Shafaei Moghaddam
> *Sent:* Wednesday, June 15, 2016 1:57 PM
> *To:* ceph-users@lists.ceph.com
> *Subject:* [ceph-users] Fio randwrite does not work on Centos 7.2 VM
>
>
>
> Hi All,
>
>
>
> Has anyone faced a similar issue? I do not have a problem with random
> read, sequential read, and sequential writes though. Everytime I try
> running fio for random writes, one osd in the cluster crashes. Here is the
> what I see at the tail of the log:
>
>
>
> ceph version 0.94.6 (e832001feaf8c176593e0325c8298e3f16dfb403)
>
> 1: ceph-osd() [0x9d6334]
>
> 2: (()+0xf100) [0x7fa2e88fb100]
>
> 3: (gsignal()+0x37) [0x7fa2e73145f7]
>
> 4: (abort()+0x148) [0x7fa2e7315ce8]
>
> 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7fa2e7c189d5]
>
> 6: (()+0x5e946) [0x7fa2e7c16946]
>
> 7: (()+0x5e973) [0x7fa2e7c16973]
>
> 8: (()+0x5eb93) [0x7fa2e7c16b93]
>
> 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x24a) [0xacd8ea]
>
> 10: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long,
> int, ThreadPool::TPHandle*)+0xa24) [0x8b8114]
>
> 11: (FileStore::_do_transactions(std::list<ObjectStore::Transaction*,
> std::allocator<ObjectStore::Transaction*> >&, unsigned long,
> ThreadPool::TPHandle*)+0x64) [0x8bcf34]
>
> 12: (FileStore::_do_op(FileStore::OpSequencer*,
> ThreadPool::TPHandle&)+0x17e) [0x8bd0ce]
>
> 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0xabe326]
>
> 14: (ThreadPool::WorkThread::entry()+0x10) [0xabf3d0]
>
> 15: (()+0x7dc5) [0x7fa2e88f3dc5]
>
> 16: (clone()+0x6d) [0x7fa2e73d528d]
>
>
>
>
> PLEASE NOTE: The information contained in this electronic mail message is
> intended only for the use of the designated recipient(s) named above. If
> the reader of this message is not the intended recipient, you are hereby
> notified that you have received this message in error and that any review,
> dissemination, distribution, or copying of this message is strictly
> prohibited. If you have received this communication in error, please notify
> the sender by telephone or e-mail (as shown above) immediately and destroy
> any and all copies of this message in your possession (whether hard copies
> or electronically stored copies).
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
