[ceph-users] Deep scrub interval not working

2018-07-04 Thread Phang WM
Hi, I want to try to distribute the deep scrub by running the deep scrub manually. I have set osd deep scrub interval to 30 days and osd scrub max interval to 14 days. However, I still getting deep-scrub even though all the last deep scrub did not pass the 30 days period. How do I reset the deep-sc

[ceph-users] Ceph slow request and rw locks

2018-06-26 Thread Phang WM
Hi, Anyone having the following issues We are randomly getting slow requests and rw locks error on the KVM volume. The slow requests eventually clear off and the cluster back to normal. The errors hit multiple osd across all 4 nodes. We have checked the following 1) Disk, all disks are ok 2) Netw

Re: [ceph-users] [Ceph-community] Ceph getting slow requests and rw locks

2018-06-26 Thread Phang WM
> > > On Wed, Jun 27, 2018 at 4:02 AM, Anthony D'Atri wrote: > Have you dumped ops-in-flight to see if the slow requests happen to > correspond to scrubs or snap trims? > > Hi Anthony, Yes, we have tried the ops-in-flight, what we get is osd_op with flag_point=delayed and event initiated, queued

Re: [ceph-users] [Ceph-community] Ceph getting slow requests and rw locks

2018-07-02 Thread Phang WM
r carried out by the disk iothread, see osd priority options instead " Does anyone have info on this osd priority setting for Luminous release? Thanks On Wed, Jun 27, 2018 at 9:47 AM, Phang WM wrote: > >> On Wed, Jun 27, 2018 at 4:02 AM, Anthony D'Atri > wrote: > &

Re: [ceph-users] [Ceph-users] Ceph getting slow requests and rw locks

2018-07-02 Thread Phang WM
Hi, After some testing, it looks like when the deep scrubbing is running, the performance is greatly affected. Even when only 1 active deep scrubbing is running. After some googling, suggestion to enable the kernel CFQ scheduler. https://ceph.com/geen-categorie/lowering-ceph-scrub-io-priority/ ce