This is quite strange. Given that you have a log, I think what you want to
do is find one request in the log, trace it through its lifetime, and see
where the time is elapsed. You may find a bifurcation, where some
categories of requests happen instantly but other categories take a second
or more; focus on the second, obviously.
-Greg
On Tue, Aug 21, 2018 at 1:31 PM Stefan Kooman <ste...@bit.nl> wrote:
> Hi,
>
> I'm trying to find out why ceph-fuse client(s) are slow. Luminous 12.2.7
> Ceph cluster, Mimic 13.2.1 ceph-fuse client. Ubuntu xenial,
> 4.13.0-38-generic kernel.
>
> Test case:
> 25 curl requests directed at a single threaded apache process (apache2
> -X).
>
> When the requests are handled by ceph-kernel client it takes about 1.5
> seconds for the first "GET /" (all fs /inode dentrie caches dropped
> before hand). Subsequent requests only take ~ 0.4 seconds. So fs caches
> seem to do their job.
>
> With ceph-fuse it's a bit different. The first and subsequent requests
> ("GET /") will take around 4-5 seconds every single time. As if the
> ceph-fuse / vfs cache does not work. ceph daemon client.id dump_cache
> shows that all dentries and inodes are in the cache ~ 589 of them. Relevant
> part of one of them:
>
> "caps": [
> {
> "mds": 0,
> "auth": 1,
> "mds": 0,
> "ino": "0x10000096a6a",
> "cap_id": 50538956,
> "issued": "pAsLsXsFscr",
> "wanted": "-",
> "seq": 1,
> "issue_seq": 1,
> "mseq": 0,
> "gen": 0
> }
> ],
> "auth_cap": 0,
> "dirty_caps": "-",
> "shared_gen": 1,
> "cache_gen": 1,
> "hold_caps_until": "0.000000",
>
> Not sure what the "issued" caps indicate. This client is currently the only
> client for this directory, so cache should not be invalidated.
> On the MDS side I hardly see any request from this client (I checked
> "dump_ops_in_flight" every second). So, I guess they should come from the
> cache. But why does it take so long? I have run ceph-fuse in debug mode
> (--debug-client=20) but this of course results in a lot of output, and I'm
> not
> sure what to look for.
>
> Watching "mds_requests" on the client every second does not show any
> request.
>
> I know the performance of ceph kernel client is (much) better than
> ceph-fuse,
> but does this also apply to objects in cache?
>
> Thanks for any hints.
>
> Gr. Stefan
>
> P.s. ceph-fuse luminous client 12.2.7 shows same result. the only active
> MDS server has 256 GB
> cache and has hardly any load. So most inodes / dentries should be cached
> there also.
>
>
> --
> | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
> | GPG: 0xD14839C6 +31 318 648 688
> <+31%20318%20648%20688> / i...@bit.nl
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
