Hi Denny,
Do not remember exactly when discards appeared in BlueStore but they are
disabled by default:
See bdev_enable_discard option.
Thanks,
Igor
On 2/15/2019 2:12 PM, Denny Kreische wrote:
Hi,
two weeks ago we upgraded one of our ceph clusters from luminous 12.2.8 to
mimic 13.2.4, cluster is SSD-only, bluestore-only, 68 nodes, 408 OSDs.
somehow we see strange behaviour since then. Single OSDs seem to block for
around 5 minutes and this causes the whole cluster and connected applications
to hang. This happened 5 times during the last 10 days at irregular times, it
didn't happen before the upgrade.
OSD log shows something like this (more log here:
https://pastebin.com/6BYam5r4):
[...]
2019-02-14 23:53:39.754 7f379a368700 -1 osd.417 340516 get_health_metrics
reporting 3 slow ops, oldest is osd_op(client.84226977.0:5112539976 0.dff
0.1d783dff (undecoded) ondisk+read+known_if_redirected e340516)
2019-02-14 23:53:40.706 7f379a368700 -1 osd.417 340516 get_health_metrics
reporting 7 slow ops, oldest is osd_op(client.84226977.0:5112539976 0.dff
0.1d783dff (undecoded) ondisk+read+known_if_redirected e340516)
[...]
In this example osd.417 seems to have a problem. I can see same log line in
other osd logs with placement groups related to osd.417.
I assume that all placement groups related to osd.417 are hanging or blocked
when osd.417 is blocked.
How can I see in detail what might cause a certain OSD to stop working?
The cluster consists of 3 different SSD vendors (micron, samsung, intel), but
only micron disks are affected until now. we earlier had problems with micron
SSDs with filestore (xfs), it was fstrim to cause single OSDs to block for
several minutes. we migrated to bluestore about a year ago. just in case, is
there any kind of ssd trim/discard happening in bluestore since mimic?
Thanks,
Denny
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
