Hi Cephers,
We have a Ceph cluster running 0.80.9, which consists of 36 OSDs with 3
replicas. Recently, some OSDs keep reporting slow request and the
cluster has a performance downgrade.
From the log of one OSD, I observe that all the slow requests are
resulted from waiting for the replicas to complete. And the replication
OSDs are not always some specific ones but could be any other two OSDs.
2016-01-06 08:17:11.887016 7f175ef25700 0 log [WRN] : slow request
1.162776 seconds old, received at 2016-01-06 08:17:11.887092:
osd_op(client.13302933.0:839452
rbd_data.c2659c728b0ddb.0000000000000024 [stat,set-alloc-hint
object_size 16777216 write_size 16777216,write 12099584~8192]
3.abd08522 ack+ondisk+write e4661) v4 currently waiting for subops
from 24,31
I dumped out the historic Ops of the OSD and noticed the following
information:
1) wait about 8 seconds for the replies from the replica OSDs.
{ "time": "2016-01-06 08:17:03.879264",
"event": "op_applied"},
{ "time": "2016-01-06 08:17:11.684598",
"event": "sub_op_applied_rec"},
{ "time": "2016-01-06 08:17:11.687016",
"event": "sub_op_commit_rec"},
2) spend more than 3 seconds in writeq and 2 seconds to write the journal.
{ "time": "2016-01-06 08:19:16.887519",
"event": "commit_queued_for_journal_write"},
{ "time": "2016-01-06 08:19:20.109339",
"event": "write_thread_in_journal_buffer"},
{ "time": "2016-01-06 08:19:22.177952",
"event": "journaled_completion_queued"},
Any ideas or suggestions?
BTW, I checked the underlying network with iperf, it works fine.
Thanks,
Jevon
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
