On Fri, Jan 25, 2019 at 9:49 PM Jonathan Woytek <woy...@dryrose.com> wrote: > > Hi friendly ceph folks. A little while after I got the message asking for > some stats, we had a network issue that caused us to take all of our > processing offline for a few hours. Since we brought everything back up, I > have been unable to duplicate the issues I was seeing. Instead, performance > of the file writer has been steady around 1.5k files/minute. Dropping the > cache causes performance to suffer. We can only get back to the 1.5k/minute > average range by restarting all of the mds daemons (well, specifically, it > looks like we can restart the first two or three, but resetting the other > three or four doesn't seem to make a difference). > > Now, I'm seeing pretty consistent sets of slow requests logged on the first > two mds daemons, stating that the slow request is a > "rejoin:client.[clientid]". When I parse the clientid's and look at the > client lists on the daemons, the clients correspond to the six swarm hosts > running the file writers. I'm attaching a small archive here of the > performance metrics Zheng asked me to produce a couple of weeks ago. I'm not > sure if they are valid for this particular problem. > > jonathan >
Looks like you have 5 active mds. I suspect your issue is related to load balancer. Please try disabling mds load balancer (add "mds_bal_max = 0" to mds section of ceph.conf). and use 'export_pin' to manually pin directories to mds (https://ceph.com/community/new-luminous-cephfs-subtree-pinning/) > > On Wed, Jan 9, 2019 at 9:10 PM Yan, Zheng <uker...@gmail.com> wrote: >> >> [...] >> Could you please run following command (for each active mds) when >> operations are fast and when operations are slow >> >> - for i in `seq 10`; do ceph daemon mds.xxx dump_historic_ops > >> mds.xxx.$i; sleep 1; done >> >> Then send the results to us >> >> Regards >> Yan, Zheng > > -- > Jonathan Woytek > http://www.dryrose.com > KB3HOZ > PGP: 462C 5F50 144D 6B09 3B65 FCE8 C1DC DEC4 E8B6 AABC _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
