I dont hear a lot of people discuss using xfs_fsr on OSDs and going over
the mailing list history it seems to have been brought up very infrequently
and never as a suggestion for regular maintenance. Perhaps its not needed.

One thing to consider trying, and to rule out something funky with the XFS
filesystem on that particular OSD/drive would be to remove the OSD entirely
from the cluster, reformat the disk, and then rebuild the OSD, putting a
brand new XFS on the OSD.

On Mon, Jan 15, 2018 at 7:36 AM, lists <li...@merit.unu.edu> wrote:

> Hi,
>
> On our three-node 24 OSDs ceph 10.2.10 cluster, we have started seeing
> slow requests on a specific OSD, during the the two-hour nightly xfs_fsr
> run from 05:00 - 07:00. This started after we applied the meltdown patches.
>
> The specific osd.10 also has the highest space utilization of all OSDs
> cluster-wide, with 45%, while the others are mostly around 40%. All OSDs
> are the same 4TB platters with journal on ssd, all with weight 1.
>
> Smart info for osd.10 shows nothing interesting I think:
>
> Current Drive Temperature:     27 C
>> Drive Trip Temperature:        60 C
>>
>> Manufactured in week 04 of year 2016
>> Specified cycle count over device lifetime:  10000
>> Accumulated start-stop cycles:  53
>> Specified load-unload count over device lifetime:  300000
>> Accumulated load-unload cycles:  697
>> Elements in grown defect list: 0
>>
>> Vendor (Seagate) cache information
>>   Blocks sent to initiator = 1933129649
>>   Blocks received from initiator = 869206640
>>   Blocks read from cache and sent to initiator = 2149311508
>>   Number of read and write commands whose size <= segment size = 676356809
>>   Number of read and write commands whose size > segment size = 12734900
>>
>> Vendor (Seagate/Hitachi) factory information
>>   number of hours powered up = 13625.88
>>   number of minutes until next internal SMART test = 8
>>
>
> Now my question:
> Could it be that osd.10 just happens to contain some data chunks that are
> heavily needed by the VMs around that time, and that the added load of an
> xfs_fsr is simply too much for it to handle?
>
> In that case, how about reweighting that osd.10 to "0", wait until all
> data has moved off osd.10, and then setting it back to "1". Would this
> result in *exactly* the same situation as before, or would it at least
> cause the data to have spread move better across the other OSDs?
>
> (with the idea that better data spread across OSDs brings also better
> distribution of load between the OSDs)
>
> Or other ideas to check out?
>
> MJ
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>



-- 
Respectfully,

Wes Dillingham
wes_dilling...@harvard.edu
Research Computing | Senior CyberInfrastructure Storage Engineer
Harvard University | 38 Oxford Street, Cambridge, Ma 02138 | Room 204
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to