You could use vmtouch to drop the specific directory from cache. It has the ability to evict directories/files from memory.
From: "negillen negillen" <negil...@gmail.com> To: "Gregory Farnum" <g...@gregs42.com> Cc: "ceph-users" <ceph-us...@ceph.com> Sent: Tuesday, June 16, 2015 7:37:23 AM Subject: Re: [ceph-users] CephFS: 'ls -alR' performance terrible unless Linux cache flushed Thanks again, even 'du' performance is terrible on node B (testing on a directory taken from Phoronix): # time du -hs /storage/test9/installed-tests/pts/pgbench-1.5.1/ 73M /storage/test9/installed-tests/pts/pgbench-1.5.1/ real 0m21.044s user 0m0.010s sys 0m0.067s Reading the files from node B doesn't seem to help with subsequent accesses in this case: # time tar c /storage/test9/installed-tests/pts/pgbench-1.5.1/>/dev/null real 1m47.650s user 0m0.041s sys 0m0.212s # time tar c /storage/test9/installed-tests/pts/pgbench-1.5.1/>/dev/null real 1m45.636s user 0m0.042s sys 0m0.214s # time ls -laR /storage/test9/installed-tests/pts/pgbench-1.5.1>/dev/null real 1m43.180s user 0m0.069s sys 0m0.236s Of course, once I dismount the CephFS on node A everything gets as fast as it can be. Am I missing something obvious here? Yes I could drop the Linux cache as a 'fix' but that would drop the entire system's cache, sounds a bit extreme! :P Unless is there a way to drop the cache only for that single dir...? On Tue, Jun 16, 2015 at 12:15 PM, Gregory Farnum < g...@gregs42.com > wrote: On Tue, Jun 16, 2015 at 12:11 PM, negillen negillen < negil...@gmail.com > wrote: > Thank you very much for your reply! > > Is there anything I can do to go around that? e.g. setting access caps to be > released after a short while? Or is there a command to manually release > access caps (so that I could run it in cron)? Well, you can drop the caches. ;) More generally, you're running into a specific hole here. If your clients are actually *accessing* the files then they should go into shared mode and this will be much faster on subsequent accesses. > This is quite a problem because we have several applications that need to > access a large number of files and when we set them to work on CephFS > latency skyrockets. What kind of shared-file access do they have? If you have a bunch of files being shared for read I'd expect this to be very fast. If different clients are writing small amounts to them in round-robin then that's unfortunately not going to work well. :( -Greg > > Thanks again and regards. > > On Tue, Jun 16, 2015 at 10:59 AM, Gregory Farnum < g...@gregs42.com > wrote: >> >> On Mon, Jun 15, 2015 at 11:34 AM, negillen negillen < negil...@gmail.com > >> wrote: >> > Hello everyone, >> > >> > something very strange is driving me crazy with CephFS (kernel driver). >> > I copy a large directory on the CephFS from one node. If I try to >> > perform a >> > 'time ls -alR' on that directory it gets executed in less than one >> > second. >> > If I try to do the same 'time ls -alR' from another node it takes >> > several >> > minutes. No matter how many times I repeat the command, the speed is >> > always >> > abysmal. The ls works fine on the node where the initial copy was >> > executed >> > from. This happens with any directory I have tried, no matter what kind >> > of >> > data is inside. >> > >> > After lots of experimenting I have found that in order to have fast ls >> > speed >> > for that dir from every node I need to flush the Linux cache on the >> > original >> > node: >> > echo 3 > /proc/sys/vm/drop_caches >> > Unmounting and remounting the CephFS on that node does the trick too. >> > >> > Anyone has a clue about what's happening here? Could this be a problem >> > with >> > the writeback fscache for the CephFS? >> > >> > Any help appreciated! Thanks and regards. :) >> >> This is a consequence of the CephFS "file capabilities" that we use to >> do distributed locking on file states. When you copy the directory on >> client A, it has full capabilities on the entire tree. When client B >> tries to do a stat on each file in that tree, it doesn't have any >> capabilities. So it sends a stat request to the MDS, which sends a cap >> update to client A requiring it to pause updates on the file and share >> its current state. Then the MDS tells client A it can keep going and >> sends the stat to client B. >> So that's: >> B -> MDS >> MDS -> A >> A -> MDS >> MDS -> B | MDS -> A >> for every file you touch. >> >> I think the particular oddity you're encountering here is that CephFS >> generally tries not to make clients drop their "exclusive" access caps >> just to satisfy a stat. If you had client B doing something with the >> files (like reading them) you would probably see different behavior. >> I'm not sure if there's something effective we can do here or not >> (it's just a bunch of heuristics when we should or should not drop >> caps), but please file a bug on the tracker ( tracker.ceph.com ) with >> this case. :) >> -Greg > > _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
