>
> On Mon, Oct 22, 2018 at 9:46 AM Dylan McCulloch <d...@unimelb.edu.au> wrote:
> >
> > On Mon, Oct 8, 2018 at 2:57 PM Dylan McCulloch <d...@unimelb.edu.au>> wrote:
> > >>
> > >> Hi all,
> > >>
> > >>
> > >> We have identified some unexpected blocking behaviour by the ceph-fs
> > >> kernel client.
> > >>
> > >>
> > >> When performing 'rm' on large files (100+GB), there appears to be a
> > >> significant delay of 10 seconds or more, before a 'stat' operation can
> > >> be performed on the same directory on the filesystem.
> > >>
> > >>
> > >> Looking at the kernel client's mds inflight-ops, we observe that there
> > >> are pending
> > >>
> > >> UNLINK operations corresponding to the deleted files.
> > >>
> > >>
> > >> We have noted some correlation between files being in the client page
> > >> cache and the blocking behaviour. For example, if the cache is dropped
> > >> or the filesystem remounted the blocking will not occur.
> > >>
> > >>
> > >> Test scenario below:
> > >>
> > >>
> > >> /mnt/cephfs_mountpoint type ceph
> > >> (rw,relatime,name=ceph_filesystem,secret=<hidden>>,noshare,acl,wsize=16777216,rasize=268439552,caps_wanted_delay_min=1,caps_wanted_delay_max=1)
> > >>
> > >>
> > >> Test1:
> > >>
> > >> 1) unmount & remount:
> > >>
> > >>
> > >> 2) Add 10 x 100GB files to a directory:
> > >>
> > >>
> > >> for i in {1..10}; do dd if=/dev/zero
> > >> of=/mnt/cephfs_mountpoint/file$i.txt count=102400 bs=1048576; done
> > >>
> > >>
> > >> 3) Delete all files in directory:
> > >>
> > >>
> > >> for i in {1..10};do rm -f /mnt/cephfs_mountpoint/file$i.txt; done
> > >>
> > >>
> > >> 4) Immediately perform ls on directory:
> > >>
> > >>
> > >> time ls /mnt/cephfs_mountpoint/test1
> > >>
> > >>
> > >> Result: delay ~16 seconds
> > >>
> > >> real 0m16.818s
> > >>
> > >> user 0m0.000s
> > >>
> > >> sys 0m0.002s
> > >>
> > >>
> >
> > > Are cephfs metadata pool and data pool on the same set of OSD. Is is
> > > possible that heavy data IO slowed down metadata IO?
> >
> > Test results are from a new pre-production cluster that does not have any
> > significant data IO. We've also confirmed the same behaviour on another
> > cluster with similar configuration. Both clusters have separate
> > device-class/crush rule for metadata pool using NVME OSDs, while the data
> > pool uses HDD OSDs.
> > Most metadata operations are unaffected. It appears that it is only
> > metadata operations on files that exist in client page cache prior to rm
> > that are delayed.
> >
>
> Ok. Please enable kernel debug when running 'ls' and send kernel log to us.
>
> echo module ceph +p > /sys/kernel/debug/dynamic_debug/control;
> time /mnt/cephfs_mountpoint/test1
> echo module ceph -p > /sys/kernel/debug/dynamic_debug/control;
>
> Yan, Zheng
Thanks Yan, Zheng
I've attached two logfiles as I ran the test twice.
The first time as previously described Test1 - cephfs_kern.log
The second time I dropped caches prior to rm as in previous Test2 -
cephfs_drop_caches_kern.log
> > >>
> > >> Test2:
> > >>
> > >>
> > >> 1) unmount & remount
> > >>
> > >>
> > >> 2) Add 10 x 100GB files to a directory
> > >>
> > >> for i in {1..10}; do dd if=/dev/zero
> > >> of=/mnt/cephfs_mountpoint/file$i.txt count=102400 bs=1048576; done
> > >>
> > >>
> > >> 3) Either a) unmount & remount; or b) drop caches
> > >>
> > >>
> > >> echo 3 >>/proc/sys/vm/drop_caches
> > >>
> > >>
> > >> 4) Delete files in directory:
> > >>
> > >>
> > >> for i in {1..10};do rm -f /mnt/cephfs_mountpoint/file$i.txt; done
> > >>
> > >>
> > >> 5) Immediately perform ls on directory:
> > >>
> > >>
> > >> time ls /mnt/cephfs_mountpoint/test1
> > >>
> > >>
> > >> Result: no delay
> > >>
> > >> real 0m0.010s
> > >>
> > >> user 0m0.000s
> > >>
> > >> sys 0m0.001s
> > >>
> > >>
> > >> Our understanding of ceph-fs’ file deletion mechanism, is that there
> > >> should be no blocking observed on the client.
> > >> http://docs.ceph.com/docs/mimic/dev/delayed-delete/ .
> > >>
> > >> It appears that if files are cached on the client, either by being
> > >> created or accessed recently it will cause the kernel client to block
> > >> for reasons we have not identified.
> > >>
> > >>
> > >> Is this a known issue, are there any ways to mitigate this behaviour?
> > >>
> > >> Our production system relies on our client’s processes having concurrent
> > >> access to the file system, and access contention must be avoided.
> > >>
> > >>
> > >> An old mailing list post that discusses changes to client’s page cache
> > >> behaviour may be relevant.
> > >>
> > >> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005692.html
> > >>
> > >>
> > >> Client System:
> > >>
> > >>
> > >> OS: RHEL7
> > >>
> > >> Kernel: 4.15.15-1
> > >>
> > >>
> > >> Cluster: Ceph: Luminous 12.2.8
> > >>
> > >>
> >
> >
> >
> > >> Thanks,
> > >>
> > >> Dylan
> > >>
> > >>
> > >> _______________________________________________
> > >> ceph-users mailing list
> > >> ceph-users@lists.ceph.com
> > >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
>
[ 1987.356149] perf-issue-part (6640): drop_caches: 3
[ 1987.394578] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1987.394580] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1987.394581] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394583] ceph: d_revalidate 00000000522328b3 'dmc' inode 00000000febc30a8 offset 0
[ 1987.394584] ceph: dentry_lease_is_valid - dentry 00000000522328b3 = 0
[ 1987.394585] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1987.394586] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394587] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 00000000522328b3 v1 = 1
[ 1987.394588] ceph: d_revalidate 00000000522328b3 valid
[ 1987.394589] ceph: dentry_lru_touch 00000000ade6eeb4 00000000522328b3 'dmc' (offset 0)
[ 1987.394590] ceph: do_getattr inode 00000000febc30a8 mask As mode 040755
[ 1987.394591] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask As)
[ 1987.394591] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394592] ceph: d_revalidate 0000000040087e94 'test1' inode 0000000070e4a810 offset 0
[ 1987.394593] ceph: dentry_lease_is_valid - dentry 0000000040087e94 = 0
[ 1987.394594] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask Fs)
[ 1987.394594] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394595] ceph: dir_lease_is_valid dir 00000000febc30a8 v1 dentry 0000000040087e94 v1 = 1
[ 1987.394595] ceph: d_revalidate 0000000040087e94 valid
[ 1987.394596] ceph: dentry_lru_touch 00000000cca4f3b4 0000000040087e94 'test1' (offset 0)
[ 1987.394597] ceph: do_getattr inode 0000000070e4a810 mask pAsLsXsFs mode 040755
[ 1987.394598] ceph: __ceph_caps_issued_mask 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx (mask pAsLsXsFs)
[ 1987.394598] ceph: __touch_cap 0000000070e4a810 cap 00000000a57451a3 mds0
[ 1987.394605] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1987.394606] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1987.394606] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394607] ceph: d_revalidate 00000000522328b3 'dmc' inode 00000000febc30a8 offset 0
[ 1987.394608] ceph: dentry_lease_is_valid - dentry 00000000522328b3 = 0
[ 1987.394608] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1987.394609] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1987.394610] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 00000000522328b3 v1 = 1
[ 1987.394610] ceph: d_revalidate 00000000522328b3 valid
[ 1987.394611] ceph: dentry_lru_touch 00000000ade6eeb4 00000000522328b3 'dmc' (offset 0)
[ 1987.394611] ceph: do_getattr inode 00000000febc30a8 mask As mode 040755
[ 1987.394612] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask As)
[ 1987.394612] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394613] ceph: d_revalidate 0000000040087e94 'test1' inode 0000000070e4a810 offset 0
[ 1987.394614] ceph: dentry_lease_is_valid - dentry 0000000040087e94 = 0
[ 1987.394614] ceph: __ceph_caps_issued_mask 00000000febc30a8 cap 00000000a0261128 issued pAsLsXsFsx (mask Fs)
[ 1987.394615] ceph: __touch_cap 00000000febc30a8 cap 00000000a0261128 mds0
[ 1987.394616] ceph: dir_lease_is_valid dir 00000000febc30a8 v1 dentry 0000000040087e94 v1 = 1
[ 1987.394616] ceph: d_revalidate 0000000040087e94 valid
[ 1987.394617] ceph: dentry_lru_touch 00000000cca4f3b4 0000000040087e94 'test1' (offset 0)
[ 1987.394617] ceph: do_getattr inode 0000000070e4a810 mask As mode 040755
[ 1987.394618] ceph: __ceph_caps_issued_mask 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx (mask As)
[ 1987.394619] ceph: __touch_cap 0000000070e4a810 cap 00000000a57451a3 mds0
[ 1987.394621] ceph: open inode 0000000070e4a810 ino 100001ed749.fffffffffffffffe file 000000008c22b0dd flags 65536 (100352)
[ 1987.394622] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394623] ceph: open 0000000070e4a810 fmode 0 want p issued pAsLsXsFsx using existing
[ 1987.394624] ceph: init_file 0000000070e4a810 000000008c22b0dd 040755 (regular)
[ 1987.394627] ceph: readdir 0000000070e4a810 file 000000008c22b0dd pos 0
[ 1987.394628] ceph: readdir off 0 -> '.'
[ 1987.394629] ceph: readdir off 1 -> '..'
[ 1987.394630] ceph: readdir fetching 100001ed749.fffffffffffffffe frag 0 offset '(null)'
[ 1987.394631] ceph: do_request on 000000009d3b260b
[ 1987.394633] ceph: reserve caps ctx=000000001d0cb3f0 need=35
[ 1987.394637] ceph: reserve caps ctx=000000001d0cb3f0 2103 = 2068 used + 35 resv + 0 avail
[ 1987.394638] ceph: __register_request 000000009d3b260b tid 124
[ 1987.394639] ceph: __choose_mds 0000000070e4a810 is_hash=1 (0) mode 2
[ 1987.394640] ceph: choose_frag(0) = 0
[ 1987.394641] ceph: choose_mds 0000000070e4a810 100001ed749.fffffffffffffffe mds0 (auth cap 00000000a57451a3)
[ 1987.394642] ceph: lookup_mds_session 00000000c8eb37bf 16
[ 1987.394643] ceph: mdsc get_session 00000000c8eb37bf 16 -> 17
[ 1987.394644] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.394645] ceph: do_request mds0 session 00000000c8eb37bf state open
[ 1987.394646] ceph: prepare_send_request 000000009d3b260b tid 124 readdir (attempt 1)
[ 1987.394647] ceph: inode 0000000070e4a810 100001ed749.fffffffffffffffe
[ 1987.394648] ceph: r_parent = (null)
[ 1987.394649] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394649] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394650] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394650] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394652] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394653] ceph: do_request waiting
[ 1987.394665] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394665] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.394725] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394726] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394731] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394732] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.394752] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.394753] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.394756] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394756] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394765] ceph: handle_caps from mds0
[ 1987.394767] ceph: op revoke ino 100001ed749.fffffffffffffffe inode 0000000070e4a810
[ 1987.394768] ceph: mds0 seq 2016 cap seq 37
[ 1987.394769] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394771] ceph: handle_cap_grant inode 0000000070e4a810 cap 00000000a57451a3 mds0 seq 37 pAsLsXsFs
[ 1987.394772] ceph: size 0 max_size 0, i_size 40
[ 1987.394772] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.394773] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.394775] ceph: my wanted = p, used = p, dirty -
[ 1987.394776] ceph: mds wanted pAsLsXsFsx -> p
[ 1987.394777] ceph: revocation: pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1987.394778] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.394779] ceph: check_caps 0000000070e4a810 file_want p used p dirty - flushing - issued pAsLsXsFs revoking Fx retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1987.394781] ceph: mds0 cap 00000000a57451a3 used p issued pAsLsXsFs implemented pAsLsXsFsx revoking Fx
[ 1987.394781] ceph: completed revocation of Fx
[ 1987.394783] ceph: __send_cap 0000000070e4a810 cap 00000000a57451a3 session 00000000c8eb37bf pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1987.394784] ceph: delaying issued pAsLsXsFs -> pAsLsXsFs, wanted pAsLsXsFsx -> p on send
[ 1987.394786] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted pAsLsXsFsx dirty - seq 37/36 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198816 xattr_len 0
[ 1987.394787] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1987.394787] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1987.394788] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1987.394789] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1987.394790] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.394791] ceph: check_caps 0000000070e4a810 file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1987.394792] ceph: __cap_set_timeouts 0000000070e4a810 min 1216 max 1216
[ 1987.394793] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655961
[ 1987.394794] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1987.394794] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1987.394801] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1987.394801] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1987.394802] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.394803] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394852] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.394853] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.394855] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.394856] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.394857] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1987.394857] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.396296] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1987.396296] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1987.396335] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396336] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396337] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396337] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396347] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396348] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396348] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396349] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396387] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396387] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396388] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396388] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396398] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396399] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396399] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396400] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396437] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396437] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396438] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396439] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396444] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396445] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396446] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396446] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396484] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396484] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396485] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396485] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.396519] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.396519] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.396520] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.396521] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397517] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.397518] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.397520] ceph: handle_caps from mds0
[ 1987.397522] ceph: op grant ino 100001ed784.fffffffffffffffe inode 00000000b9dd47ac
[ 1987.397523] ceph: mds0 seq 2017 cap seq 92
[ 1987.397524] ceph: __ceph_caps_issued 00000000b9dd47ac cap 0000000024c864e1 issued p
[ 1987.397525] ceph: handle_cap_grant inode 00000000b9dd47ac cap 0000000024c864e1 mds0 seq 92 pAsLsXsFscr
[ 1987.397526] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397526] ceph: __ceph_caps_issued 00000000b9dd47ac cap 0000000024c864e1 issued p
[ 1987.397527] ceph: 00000000b9dd47ac mode 0100644 uid.gid 0.0
[ 1987.397529] ceph: my wanted = -, used = -, dirty -
[ 1987.397530] ceph: mds wanted Ls -> -
[ 1987.397531] ceph: grant: p -> pAsLsXsFscr
[ 1987.397531] ceph: invalidate_aliases inode 00000000b9dd47ac
[ 1987.397532] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.397533] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397534] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1987.397535] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1987.397536] ceph: handle_reply 00000000ce76d613
[ 1987.397537] ceph: __unregister_request 00000000ce76d613 tid 119
[ 1987.397538] ceph: got safe reply 119, mds0
[ 1987.397543] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1987.397544] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1987.397546] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1987.397547] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1987.397548] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1987.397548] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1987.397550] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397551] ceph: ceph_d_prune 00000000062c3ffd
[ 1987.397554] ceph: destroy_inode 00000000b9dd47ac ino 100001ed784.fffffffffffffffe
[ 1987.397555] ceph: __ceph_remove_cap 0000000024c864e1 from 00000000b9dd47ac
[ 1987.397556] ceph: put_snap_realm 1 000000008b101794 35 -> 34
[ 1987.397557] ceph: __cap_delay_cancel 00000000b9dd47ac
[ 1987.397558] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397559] ceph: d_release 00000000062c3ffd
[ 1987.397560] ceph: dentry_lru_del 00000000ee10f333 00000000062c3ffd 'file1.txt'
[ 1987.397561] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1987.397562] ceph: unreserve caps ctx=00000000f34ca0f9 count=0
[ 1987.397563] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397564] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397565] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.397565] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1987.397567] ceph: handle_caps from mds0
[ 1987.397568] ceph: op grant ino 100001ed785.fffffffffffffffe inode 00000000150130f5
[ 1987.397568] ceph: mds0 seq 2018 cap seq 83
[ 1987.397569] ceph: __ceph_caps_issued 00000000150130f5 cap 0000000061a546e6 issued p
[ 1987.397570] ceph: handle_cap_grant inode 00000000150130f5 cap 0000000061a546e6 mds0 seq 83 pAsLsXsFscr
[ 1987.397571] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397571] ceph: __ceph_caps_issued 00000000150130f5 cap 0000000061a546e6 issued p
[ 1987.397572] ceph: 00000000150130f5 mode 0100644 uid.gid 0.0
[ 1987.397573] ceph: my wanted = -, used = -, dirty -
[ 1987.397573] ceph: mds wanted Ls -> -
[ 1987.397574] ceph: grant: p -> pAsLsXsFscr
[ 1987.397574] ceph: invalidate_aliases inode 00000000150130f5
[ 1987.397575] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397576] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397577] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1987.397578] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1987.397579] ceph: handle_reply 00000000b5a6a4b6
[ 1987.397579] ceph: __unregister_request 00000000b5a6a4b6 tid 120
[ 1987.397580] ceph: got safe reply 120, mds0
[ 1987.397581] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1987.397581] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1987.397583] ceph: mdsc con_put 00000000c8eb37bf (16)
[ 1987.397583] ceph: mdsc put_session 00000000c8eb37bf 17 -> 16
[ 1987.397584] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397585] ceph: ceph_d_prune 000000003a9fb321
[ 1987.397586] ceph: destroy_inode 00000000150130f5 ino 100001ed785.fffffffffffffffe
[ 1987.397586] ceph: __ceph_remove_cap 0000000061a546e6 from 00000000150130f5
[ 1987.397587] ceph: put_snap_realm 1 000000008b101794 34 -> 33
[ 1987.397588] ceph: __cap_delay_cancel 00000000150130f5
[ 1987.397588] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397589] ceph: d_release 000000003a9fb321
[ 1987.397589] ceph: dentry_lru_del 000000009f8c7d76 000000003a9fb321 'file2.txt'
[ 1987.397590] ceph: mdsc put_session 00000000c8eb37bf 16 -> 15
[ 1987.397591] ceph: unreserve caps ctx=0000000060eb8e9b count=0
[ 1987.397591] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397592] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397593] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1987.397594] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1987.397595] ceph: handle_caps from mds0
[ 1987.397596] ceph: op grant ino 100001ed786.fffffffffffffffe inode 0000000035bcecb2
[ 1987.397596] ceph: mds0 seq 2019 cap seq 80
[ 1987.397597] ceph: __ceph_caps_issued 0000000035bcecb2 cap 00000000d1fcbbd9 issued p
[ 1987.397598] ceph: handle_cap_grant inode 0000000035bcecb2 cap 00000000d1fcbbd9 mds0 seq 80 pAsLsXsFscr
[ 1987.397598] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397599] ceph: __ceph_caps_issued 0000000035bcecb2 cap 00000000d1fcbbd9 issued p
[ 1987.397600] ceph: 0000000035bcecb2 mode 0100644 uid.gid 0.0
[ 1987.397601] ceph: my wanted = -, used = -, dirty -
[ 1987.397601] ceph: mds wanted Ls -> -
[ 1987.397602] ceph: grant: p -> pAsLsXsFscr
[ 1987.397602] ceph: invalidate_aliases inode 0000000035bcecb2
[ 1987.397603] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397604] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397605] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1987.397605] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1987.397606] ceph: handle_reply 000000007df1685a
[ 1987.397607] ceph: __unregister_request 000000007df1685a tid 121
[ 1987.397608] ceph: got safe reply 121, mds0
[ 1987.397609] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1987.397609] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1987.397610] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1987.397611] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1987.397612] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397612] ceph: ceph_d_prune 000000004fa570bf
[ 1987.397613] ceph: destroy_inode 0000000035bcecb2 ino 100001ed786.fffffffffffffffe
[ 1987.397614] ceph: __ceph_remove_cap 00000000d1fcbbd9 from 0000000035bcecb2
[ 1987.397615] ceph: put_snap_realm 1 000000008b101794 33 -> 32
[ 1987.397615] ceph: __cap_delay_cancel 0000000035bcecb2
[ 1987.397616] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397616] ceph: d_release 000000004fa570bf
[ 1987.397617] ceph: dentry_lru_del 0000000055b76e3b 000000004fa570bf 'file3.txt'
[ 1987.397618] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1987.397618] ceph: unreserve caps ctx=0000000065473d82 count=0
[ 1987.397619] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397619] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397620] ceph: mdsc get_session 00000000c8eb37bf 11 -> 12
[ 1987.397621] ceph: mdsc con_get 00000000c8eb37bf ok (12)
[ 1987.397622] ceph: handle_caps from mds0
[ 1987.397623] ceph: op grant ino 100001ed787.fffffffffffffffe inode 000000005e234bc9
[ 1987.397624] ceph: mds0 seq 2020 cap seq 83
[ 1987.397624] ceph: __ceph_caps_issued 000000005e234bc9 cap 0000000053d84a0b issued p
[ 1987.397625] ceph: handle_cap_grant inode 000000005e234bc9 cap 0000000053d84a0b mds0 seq 83 pAsLsXsFscr
[ 1987.397626] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397626] ceph: __ceph_caps_issued 000000005e234bc9 cap 0000000053d84a0b issued p
[ 1987.397627] ceph: 000000005e234bc9 mode 0100644 uid.gid 0.0
[ 1987.397627] ceph: my wanted = -, used = -, dirty -
[ 1987.397628] ceph: mds wanted Ls -> -
[ 1987.397628] ceph: grant: p -> pAsLsXsFscr
[ 1987.397629] ceph: invalidate_aliases inode 000000005e234bc9
[ 1987.397629] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397630] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397631] ceph: mdsc get_session 00000000c8eb37bf 11 -> 12
[ 1987.397632] ceph: mdsc con_get 00000000c8eb37bf ok (12)
[ 1987.397633] ceph: handle_reply 00000000d74a8455
[ 1987.397633] ceph: __unregister_request 00000000d74a8455 tid 122
[ 1987.397634] ceph: got safe reply 122, mds0
[ 1987.397635] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1987.397635] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1987.397637] ceph: mdsc con_put 00000000c8eb37bf (10)
[ 1987.397637] ceph: mdsc put_session 00000000c8eb37bf 11 -> 10
[ 1987.397638] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397638] ceph: ceph_d_prune 00000000298dfd11
[ 1987.397639] ceph: destroy_inode 000000005e234bc9 ino 100001ed787.fffffffffffffffe
[ 1987.397640] ceph: __ceph_remove_cap 0000000053d84a0b from 000000005e234bc9
[ 1987.397641] ceph: put_snap_realm 1 000000008b101794 32 -> 31
[ 1987.397641] ceph: __cap_delay_cancel 000000005e234bc9
[ 1987.397642] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397642] ceph: d_release 00000000298dfd11
[ 1987.397643] ceph: dentry_lru_del 00000000fc7a978f 00000000298dfd11 'file4.txt'
[ 1987.397643] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1987.397644] ceph: unreserve caps ctx=00000000c9ad5916 count=0
[ 1987.397644] ceph: mdsc con_put 00000000c8eb37bf (8)
[ 1987.397645] ceph: mdsc put_session 00000000c8eb37bf 9 -> 8
[ 1987.397646] ceph: mdsc get_session 00000000c8eb37bf 8 -> 9
[ 1987.397647] ceph: mdsc con_get 00000000c8eb37bf ok (9)
[ 1987.397648] ceph: handle_reply 000000009d3b260b
[ 1987.397649] ceph: __unregister_request 000000009d3b260b tid 124
[ 1987.397649] ceph: handle_reply tid 124 result 0
[ 1987.397650] ceph: update_snap_trace deletion=0
[ 1987.397651] ceph: lookup_snap_realm 1 000000008b101794
[ 1987.397652] ceph: get_realm 000000008b101794 31 -> 32
[ 1987.397653] ceph: update_snap_trace 1 000000008b101794 seq 1 unchanged
[ 1987.397654] ceph: done with 1 000000008b101794, invalidated=0, 00000000edc90cda 00000000edc90cda
[ 1987.397655] ceph: fill_trace 000000009d3b260b is_dentry 0 is_target 1
[ 1987.397657] ceph: get_inode on 1099513648969=100001ed749.fffffffffffffffe got 0000000070e4a810
[ 1987.397658] ceph: fill_inode 0000000070e4a810 ino 100001ed749.fffffffffffffffe v 9650292 had 9650273
[ 1987.397659] ceph: get_cap ctx=000000001d0cb3f0 (35) 2103 = 2068 used + 35 resv + 0 avail
[ 1987.397660] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397661] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.397663] ceph: add_cap 0000000070e4a810 mds0 cap 37bd31 pAsLsXsFs seq 38
[ 1987.397663] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397664] ceph: issued pAsLsXsFs, mds wanted pAsLsXsFsx, actual p, queueing
[ 1987.397665] ceph: __cap_set_timeouts 0000000070e4a810 min 1213 max 1213
[ 1987.397666] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655961
[ 1987.397667] ceph: add_cap inode 0000000070e4a810 (100001ed749.fffffffffffffffe) cap 00000000a57451a3 pAsLsXsFs now pAsLsXsFs seq 38 mds0
[ 1987.397668] ceph: put_cap 000000000df0548c 2103 = 2069 used + 34 resv + 0 avail
[ 1987.397669] ceph: fill_trace done err=0
[ 1987.397670] ceph: readdir_prepopulate 0 items under dn 0000000040087e94
[ 1987.397671] ceph: readdir_prepopulate done
[ 1987.397671] ceph: unreserve caps ctx=000000001d0cb3f0 count=34
[ 1987.397672] ceph: unreserve caps 2103 = 2068 used + 0 resv + 35 avail
[ 1987.397673] ceph: put_snap_realm 1 000000008b101794 32 -> 31
[ 1987.397676] ceph: mdsc get_session 00000000c8eb37bf 9 -> 10
[ 1987.397676] ceph: mdsc con_get 00000000c8eb37bf ok (10)
[ 1987.397677] ceph: do_request waited, got 0
[ 1987.397678] ceph: do_request 000000009d3b260b done, result 0
[ 1987.397679] ceph: readdir got and parsed readdir result=0 on frag 0, end=1, complete=1, hash_order=1
[ 1987.397679] ceph: handle_caps from mds0
[ 1987.397680] ceph: op grant ino 100001ed749.fffffffffffffffe inode 0000000070e4a810
[ 1987.397681] ceph: mds0 seq 2021 cap seq 39
[ 1987.397682] ceph: readdir frag 0 num 0 pos 2 chunk first 0
[ 1987.397683] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397683] ceph: mdsc con_put 00000000c8eb37bf (9)
[ 1987.397684] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1987.397685] ceph: handle_cap_grant inode 0000000070e4a810 cap 00000000a57451a3 mds0 seq 39 pAsLsXsFsx
[ 1987.397686] ceph: size 0 max_size 0, i_size 40
[ 1987.397687] ceph: mdsc con_put 00000000c8eb37bf (8)
[ 1987.397688] ceph: mdsc put_session 00000000c8eb37bf 9 -> 8
[ 1987.397688] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFs
[ 1987.397690] ceph: 0000000070e4a810 mode 040755 uid.gid 0.0
[ 1987.397690] ceph: my wanted = p, used = p, dirty -
[ 1987.397691] ceph: mds wanted pAsLsXsFsx -> p
[ 1987.397691] ceph: grant: pAsLsXsFs -> pAsLsXsFsx
[ 1987.397692] ceph: mdsc con_put 00000000c8eb37bf (7)
[ 1987.397693] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397694] ceph: mdsc put_session 00000000c8eb37bf 8 -> 7
[ 1987.397695] ceph: mdsc put_session 00000000c8eb37bf 8 -> 7
[ 1987.397695] ceph: unreserve caps ctx=000000001d0cb3f0 count=0
[ 1987.397697] ceph: mdsc get_session 00000000c8eb37bf 6 -> 7
[ 1987.397698] ceph: marking 0000000070e4a810 complete and ordered
[ 1987.397698] ceph: readdir 0000000070e4a810 file 000000008c22b0dd done.
[ 1987.397699] ceph: mdsc con_get 00000000c8eb37bf ok (7)
[ 1987.397701] ceph: handle_caps from mds0
[ 1987.397702] ceph: op grant ino 100001ed788.fffffffffffffffe inode 000000000be62518
[ 1987.397702] ceph: readdir 0000000070e4a810 file 000000008c22b0dd pos 2
[ 1987.397703] ceph: mds0 seq 2022 cap seq 84
[ 1987.397704] ceph: __ceph_caps_issued 000000000be62518 cap 0000000070d8cfca issued p
[ 1987.397705] ceph: handle_cap_grant inode 000000000be62518 cap 0000000070d8cfca mds0 seq 84 pAsLsXsFscr
[ 1987.397706] ceph: release inode 0000000070e4a810 file 000000008c22b0dd
[ 1987.397707] ceph: put_fmode 0000000070e4a810 fmode 0 {0,0,0,0}
[ 1987.397708] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1987.397709] ceph: __ceph_caps_issued 000000000be62518 cap 0000000070d8cfca issued p
[ 1987.397709] ceph: __ceph_caps_issued 0000000070e4a810 cap 00000000a57451a3 issued pAsLsXsFsx
[ 1987.397711] ceph: check_caps 0000000070e4a810 file_want - used p dirty - flushing - issued pAsLsXsFsx revoking - retain pAsLsXsFsx
[ 1987.397712] ceph: 000000000be62518 mode 0100644 uid.gid 0.0
[ 1987.397713] ceph: my wanted = -, used = -, dirty -
[ 1987.397713] ceph: mds0 cap 00000000a57451a3 used p issued pAsLsXsFsx implemented pAsLsXsFsx revoking -
[ 1987.397714] ceph: delaying issued pAsLsXsFsx -> pAsLsXsFsx, wanted pAsLsXsFsx -> AsLsXsFsx
[ 1987.397715] ceph: mds wanted Ls -> -
[ 1987.397716] ceph: __cap_set_timeouts 0000000070e4a810 min 1207 max 1207
[ 1987.397717] ceph: __cap_delay_requeue 0000000070e4a810 flags 0 at 4296655955
[ 1987.397718] ceph: grant: p -> pAsLsXsFscr
[ 1987.397719] ceph: invalidate_aliases inode 000000000be62518
[ 1987.397719] ceph: mdsc con_put 00000000c8eb37bf (6)
[ 1987.397720] ceph: mdsc put_session 00000000c8eb37bf 7 -> 6
[ 1987.397721] ceph: mdsc get_session 00000000c8eb37bf 6 -> 7
[ 1987.397722] ceph: mdsc con_get 00000000c8eb37bf ok (7)
[ 1987.397723] ceph: handle_reply 000000006fb8dc63
[ 1987.397724] ceph: __unregister_request 000000006fb8dc63 tid 123
[ 1987.397724] ceph: got safe reply 123, mds0
[ 1987.397725] ceph: mdsc con_put 00000000c8eb37bf (6)
[ 1987.397726] ceph: mdsc put_session 00000000c8eb37bf 7 -> 6
[ 1987.397727] ceph: mdsc con_put 00000000c8eb37bf (5)
[ 1987.397727] ceph: mdsc put_session 00000000c8eb37bf 6 -> 5
[ 1987.397728] ceph: put_cap_refs 0000000070e4a810 had p
[ 1987.397729] ceph: ceph_d_prune 000000006d799af9
[ 1987.397735] ceph: destroy_inode 000000000be62518 ino 100001ed788.fffffffffffffffe
[ 1987.397735] ceph: __ceph_remove_cap 0000000070d8cfca from 000000000be62518
[ 1987.397736] ceph: put_snap_realm 1 000000008b101794 31 -> 30
[ 1987.397736] ceph: __cap_delay_cancel 000000000be62518
[ 1987.397737] ceph: __ceph_destroy_xattrs p= (null)
[ 1987.397737] ceph: d_release 000000006d799af9
[ 1987.397738] ceph: dentry_lru_del 00000000fa1e88d3 000000006d799af9 'file5.txt'
[ 1987.397739] ceph: mdsc put_session 00000000c8eb37bf 5 -> 4
[ 1987.397740] ceph: unreserve caps ctx=000000005cb10386 count=0
[ 1987.397740] ceph: mdsc con_put 00000000c8eb37bf (3)
[ 1987.397741] ceph: mdsc put_session 00000000c8eb37bf 4 -> 3
[ 1987.397744] ceph: mdsc con_put 00000000c8eb37bf (2)
[ 1987.397744] ceph: mdsc put_session 00000000c8eb37bf 3 -> 2
[ 1987.397745] ceph: mdsc con_put 00000000c8eb37bf (1)
[ 1987.397745] ceph: mdsc put_session 00000000c8eb37bf 2 -> 1
[ 1280.529935] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1280.529938] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1280.529939] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529942] ceph: d_revalidate 0000000006dc6eec 'dmc' inode 000000006611481f offset 1148534478444953602
[ 1280.529943] ceph: dentry_lease_is_valid - dentry 0000000006dc6eec = 0
[ 1280.529944] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1280.529944] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529945] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 0000000006dc6eec v1 = 1
[ 1280.529946] ceph: d_revalidate 0000000006dc6eec valid
[ 1280.529948] ceph: dentry_lru_touch 00000000d589fb13 0000000006dc6eec 'dmc' (offset 1148534478444953602)
[ 1280.529949] ceph: do_getattr inode 000000006611481f mask As mode 040755
[ 1280.529950] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask As)
[ 1280.529951] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529952] ceph: d_revalidate 0000000010f74030 'test1' inode 000000007dbf994a offset 1152287136058179586
[ 1280.529953] ceph: dentry_lease_is_valid - dentry 0000000010f74030 = 0
[ 1280.529953] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask Fs)
[ 1280.529954] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529955] ceph: dir_lease_is_valid dir 000000006611481f v1 dentry 0000000010f74030 v1 = 1
[ 1280.529956] ceph: d_revalidate 0000000010f74030 valid
[ 1280.529957] ceph: dentry_lru_touch 00000000bcae6752 0000000010f74030 'test1' (offset 1152287136058179586)
[ 1280.529958] ceph: do_getattr inode 000000007dbf994a mask pAsLsXsFs mode 040755
[ 1280.529959] ceph: __ceph_caps_issued_mask 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx (mask pAsLsXsFs)
[ 1280.529959] ceph: __touch_cap 000000007dbf994a cap 000000001044692e mds0
[ 1280.529967] ceph: do_getattr inode 000000005a6fde97 mask As mode 040775
[ 1280.529968] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask As)
[ 1280.529969] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529969] ceph: d_revalidate 0000000006dc6eec 'dmc' inode 000000006611481f offset 1148534478444953602
[ 1280.529970] ceph: dentry_lease_is_valid - dentry 0000000006dc6eec = 0
[ 1280.529971] ceph: __ceph_caps_issued_mask 000000005a6fde97 cap 00000000ca9565a0 issued pAsLsXsFs (mask Fs)
[ 1280.529972] ceph: __touch_cap 000000005a6fde97 cap 00000000ca9565a0 mds0
[ 1280.529972] ceph: dir_lease_is_valid dir 000000005a6fde97 v1 dentry 0000000006dc6eec v1 = 1
[ 1280.529973] ceph: d_revalidate 0000000006dc6eec valid
[ 1280.529974] ceph: dentry_lru_touch 00000000d589fb13 0000000006dc6eec 'dmc' (offset 1148534478444953602)
[ 1280.529974] ceph: do_getattr inode 000000006611481f mask As mode 040755
[ 1280.529975] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask As)
[ 1280.529976] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529977] ceph: d_revalidate 0000000010f74030 'test1' inode 000000007dbf994a offset 1152287136058179586
[ 1280.529977] ceph: dentry_lease_is_valid - dentry 0000000010f74030 = 0
[ 1280.529978] ceph: __ceph_caps_issued_mask 000000006611481f cap 000000003a118540 issued pAsLsXsFsx (mask Fs)
[ 1280.529978] ceph: __touch_cap 000000006611481f cap 000000003a118540 mds0
[ 1280.529979] ceph: dir_lease_is_valid dir 000000006611481f v1 dentry 0000000010f74030 v1 = 1
[ 1280.529980] ceph: d_revalidate 0000000010f74030 valid
[ 1280.529980] ceph: dentry_lru_touch 00000000bcae6752 0000000010f74030 'test1' (offset 1152287136058179586)
[ 1280.529981] ceph: do_getattr inode 000000007dbf994a mask As mode 040755
[ 1280.529982] ceph: __ceph_caps_issued_mask 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx (mask As)
[ 1280.529983] ceph: __touch_cap 000000007dbf994a cap 000000001044692e mds0
[ 1280.529984] ceph: open inode 000000007dbf994a ino 100001ed749.fffffffffffffffe file 00000000fa762655 flags 65536 (100352)
[ 1280.529986] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.529987] ceph: open 000000007dbf994a fmode 0 want p issued pAsLsXsFsx using existing
[ 1280.529988] ceph: init_file 000000007dbf994a 00000000fa762655 040755 (regular)
[ 1280.529992] ceph: readdir 000000007dbf994a file 00000000fa762655 pos 0
[ 1280.529993] ceph: readdir off 0 -> '.'
[ 1280.529993] ceph: readdir off 1 -> '..'
[ 1280.529994] ceph: readdir fetching 100001ed749.fffffffffffffffe frag 0 offset '(null)'
[ 1280.529996] ceph: do_request on 000000004f565148
[ 1280.529998] ceph: reserve caps ctx=000000008187b867 need=35
[ 1280.529999] ceph: reserve caps ctx=000000008187b867 2096 = 2061 used + 35 resv + 0 avail
[ 1280.530000] ceph: __register_request 000000004f565148 tid 110
[ 1280.530002] ceph: __choose_mds 000000007dbf994a is_hash=1 (0) mode 2
[ 1280.530003] ceph: choose_frag(0) = 0
[ 1280.530004] ceph: choose_mds 000000007dbf994a 100001ed749.fffffffffffffffe mds0 (auth cap 000000001044692e)
[ 1280.530005] ceph: lookup_mds_session 00000000c8eb37bf 16
[ 1280.530006] ceph: mdsc get_session 00000000c8eb37bf 16 -> 17
[ 1280.530007] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1280.530008] ceph: do_request mds0 session 00000000c8eb37bf state open
[ 1280.530009] ceph: prepare_send_request 000000004f565148 tid 110 readdir (attempt 1)
[ 1280.530010] ceph: inode 000000007dbf994a 100001ed749.fffffffffffffffe
[ 1280.530012] ceph: r_parent = (null)
[ 1280.530012] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.530013] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.530014] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.530015] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.530018] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.530019] ceph: do_request waiting
[ 1280.530040] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.530040] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.530084] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.530091] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.530120] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.530121] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.530122] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.530123] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531194] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531195] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531205] ceph: handle_caps from mds0
[ 1280.531206] ceph: op revoke ino 100001ed749.fffffffffffffffe inode 000000007dbf994a
[ 1280.531207] ceph: mds0 seq 1198 cap seq 20
[ 1280.531208] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.531209] ceph: handle_cap_grant inode 000000007dbf994a cap 000000001044692e mds0 seq 20 pAsLsXsFs
[ 1280.531210] ceph: size 0 max_size 0, i_size 40
[ 1280.531211] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1280.531211] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1280.531213] ceph: my wanted = p, used = p, dirty -
[ 1280.531213] ceph: mds wanted pAsLsXsFsx -> p
[ 1280.531215] ceph: revocation: pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1280.531215] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1280.531217] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking Fx retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1280.531218] ceph: mds0 cap 000000001044692e used p issued pAsLsXsFs implemented pAsLsXsFsx revoking Fx
[ 1280.531219] ceph: completed revocation of Fx
[ 1280.531220] ceph: __send_cap 000000007dbf994a cap 000000001044692e session 00000000c8eb37bf pAsLsXsFsx -> pAsLsXsFs (revoking Fx)
[ 1280.531221] ceph: delaying issued pAsLsXsFs -> pAsLsXsFs, wanted pAsLsXsFsx -> p on send
[ 1280.531223] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted pAsLsXsFsx dirty - seq 20/19 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198816 xattr_len 0
[ 1280.531224] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1280.531224] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1280.531225] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1280.531225] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1280.531226] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1280.531227] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl AUTHONLY NODELAY
[ 1280.531228] ceph: __cap_set_timeouts 000000007dbf994a min 1082 max 1082
[ 1280.531229] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295948943
[ 1280.531230] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1280.531230] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1280.531236] ceph: mdsc con_put 00000000c8eb37bf (20)
[ 1280.531237] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1280.531239] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531239] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531297] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531298] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531300] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531301] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531302] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.531302] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.531946] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.531947] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.531949] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531950] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.531954] ceph: handle_caps from mds0
[ 1280.531957] ceph: op grant ino 100001ed77f.fffffffffffffffe inode 00000000fdca32d0
[ 1280.531958] ceph: mds0 seq 1199 cap seq 114
[ 1280.531958] ceph: __ceph_caps_issued 00000000fdca32d0 cap 0000000054a99657 issued pFc
[ 1280.531959] ceph: handle_cap_grant inode 00000000fdca32d0 cap 0000000054a99657 mds0 seq 114 pAsLsXsFscr
[ 1280.531960] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1280.531960] ceph: __ceph_caps_issued 00000000fdca32d0 cap 0000000054a99657 issued pFc
[ 1280.531961] ceph: 00000000fdca32d0 mode 0100644 uid.gid 0.0
[ 1280.531962] ceph: my wanted = Fc, used = Fc, dirty -
[ 1280.531963] ceph: mds wanted LsFc -> Fc
[ 1280.531963] ceph: grant: pFc -> pAsLsXsFscr
[ 1280.531964] ceph: invalidate_aliases inode 00000000fdca32d0
[ 1280.531965] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.531965] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.531967] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.531968] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.531995] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.531995] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.531999] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.531999] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532000] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532000] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532006] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532006] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532007] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532007] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532012] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532012] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532013] ceph: handle_reply 00000000de20b748
[ 1280.532014] ceph: __unregister_request 00000000de20b748 tid 105
[ 1280.532015] ceph: got safe reply 105, mds0
[ 1280.532016] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532017] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532017] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1280.532018] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1280.532019] ceph: put_cap_refs 000000007dbf994a had p
[ 1280.532020] ceph: ceph_d_prune 00000000c2665322
[ 1280.532042] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1280.532043] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1280.532187] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532187] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532188] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532188] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532190] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532191] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532191] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532192] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532194] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532195] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532195] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532196] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532198] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532198] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532199] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532199] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532202] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532202] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532203] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532203] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.532205] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.532206] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1280.532206] ceph: mdsc con_put 00000000c8eb37bf (19)
[ 1280.532207] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1280.957403] ceph: mdsc delayed_work
[ 1280.957404] ceph: check_delayed_caps
[ 1280.957405] ceph: lookup_mds_session 00000000c8eb37bf 19
[ 1280.957406] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1280.957408] ceph: send_renew_caps to mds0 (up:active)
[ 1280.957409] ceph: mdsc get_session 00000000c8eb37bf 20 -> 21
[ 1280.957410] ceph: mdsc con_get 00000000c8eb37bf ok (21)
[ 1280.957410] ceph: mdsc get_session 00000000c8eb37bf 21 -> 22
[ 1280.957411] ceph: mdsc con_get 00000000c8eb37bf ok (22)
[ 1280.957412] ceph: mdsc con_put 00000000c8eb37bf (21)
[ 1280.957412] ceph: mdsc put_session 00000000c8eb37bf 22 -> 21
[ 1280.957413] ceph: mdsc put_session 00000000c8eb37bf 21 -> 20
[ 1284.592212] ceph: destroy_inode 00000000fdca32d0 ino 100001ed77f.fffffffffffffffe
[ 1284.592214] ceph: __ceph_remove_cap 0000000054a99657 from 00000000fdca32d0
[ 1284.592216] ceph: put_snap_realm 1 000000008b101794 2061 -> 2060
[ 1284.592217] ceph: __cap_delay_cancel 00000000fdca32d0
[ 1284.592218] ceph: __ceph_destroy_xattrs p= (null)
[ 1284.592219] ceph: d_release 00000000c2665322
[ 1284.592221] ceph: dentry_lru_del 00000000c348e53a 00000000c2665322 'file1.txt'
[ 1284.592223] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1284.592224] ceph: unreserve caps ctx=00000000134ad6f9 count=0
[ 1284.592226] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592227] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592234] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1284.592235] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1284.592240] ceph: handle_caps from mds0
[ 1284.592242] ceph: op grant ino 100001ed780.fffffffffffffffe inode 00000000bef38140
[ 1284.592243] ceph: mds0 seq 1200 cap seq 86
[ 1284.592245] ceph: __ceph_caps_issued 00000000bef38140 cap 0000000019e816a1 issued pFc
[ 1284.592246] ceph: handle_cap_grant inode 00000000bef38140 cap 0000000019e816a1 mds0 seq 86 pAsLsXsFscr
[ 1284.592247] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1284.592248] ceph: __ceph_caps_issued 00000000bef38140 cap 0000000019e816a1 issued pFc
[ 1284.592249] ceph: 00000000bef38140 mode 0100644 uid.gid 0.0
[ 1284.592251] ceph: my wanted = Fc, used = Fc, dirty -
[ 1284.592252] ceph: mds wanted LsFc -> Fc
[ 1284.592253] ceph: grant: pFc -> pAsLsXsFscr
[ 1284.592253] ceph: invalidate_aliases inode 00000000bef38140
[ 1284.592254] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592255] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592256] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1284.592257] ceph: mdsc con_get 00000000c8eb37bf ok (19)
[ 1284.592259] ceph: handle_reply 00000000db0c7bcc
[ 1284.592259] ceph: __unregister_request 00000000db0c7bcc tid 106
[ 1284.592260] ceph: got safe reply 106, mds0
[ 1284.592261] ceph: mdsc con_put 00000000c8eb37bf (18)
[ 1284.592262] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1284.592263] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1284.592263] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1284.592264] ceph: put_cap_refs 000000007dbf994a had p
[ 1284.592265] ceph: ceph_d_prune 0000000063d6dbc8
[ 1287.101272] ceph: mdsc delayed_work
[ 1287.101274] ceph: check_delayed_caps
[ 1287.101275] ceph: check_delayed_caps on 000000007dbf994a
[ 1287.101279] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1287.101280] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl NODELAY
[ 1287.101282] ceph: mds0 cap 000000001044692e used p issued pAsLsXsFs implemented pAsLsXsFs revoking -
[ 1287.101284] ceph: __send_cap 000000007dbf994a cap 000000001044692e session 00000000c8eb37bf pAsLsXsFs -> pAsLsXsFs (revoking -)
[ 1287.101287] ceph: send_cap_msg update 37bd31 100001ed749 caps pAsLsXsFs wanted p dirty - seq 20/19 tid 0/0 mseq 0 follows 0 size 40/0 xattr_ver 18446744072109198672 xattr_len 0
[ 1287.101289] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1287.101290] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1287.101291] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1287.101292] ceph: check_caps 000000007dbf994a file_want p used p dirty - flushing - issued pAsLsXsFs revoking - retain pAsxLsxXsxFsxcrwbl NODELAY
[ 1287.101293] ceph: lookup_mds_session 00000000c8eb37bf 18
[ 1287.101293] ceph: mdsc get_session 00000000c8eb37bf 18 -> 19
[ 1287.101295] ceph: put_cap 0000000054a99657 2096 = 2061 used + 35 resv + 0 avail
[ 1287.101296] ceph: send_cap_releases mds0 00000000f1f253c8
[ 1287.101297] ceph: mdsc get_session 00000000c8eb37bf 19 -> 20
[ 1287.101297] ceph: mdsc con_get 00000000c8eb37bf ok (20)
[ 1287.101298] ceph: mdsc put_session 00000000c8eb37bf 20 -> 19
[ 1288.648118] ceph: destroy_inode 00000000bef38140 ino 100001ed780.fffffffffffffffe
[ 1288.648121] ceph: __ceph_remove_cap 0000000019e816a1 from 00000000bef38140
[ 1288.648123] ceph: put_snap_realm 1 000000008b101794 2060 -> 2059
[ 1288.648124] ceph: __cap_delay_cancel 00000000bef38140
[ 1288.648125] ceph: __ceph_destroy_xattrs p= (null)
[ 1288.648126] ceph: d_release 0000000063d6dbc8
[ 1288.648128] ceph: dentry_lru_del 0000000041d9b9ba 0000000063d6dbc8 'file2.txt'
[ 1288.648130] ceph: mdsc put_session 00000000c8eb37bf 19 -> 18
[ 1288.648131] ceph: unreserve caps ctx=0000000023d04527 count=0
[ 1288.648133] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648133] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648139] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1288.648140] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1288.648144] ceph: handle_caps from mds0
[ 1288.648147] ceph: op grant ino 100001ed781.fffffffffffffffe inode 00000000211593be
[ 1288.648148] ceph: mds0 seq 1201 cap seq 81
[ 1288.648149] ceph: __ceph_caps_issued 00000000211593be cap 00000000651e8193 issued pFc
[ 1288.648151] ceph: handle_cap_grant inode 00000000211593be cap 00000000651e8193 mds0 seq 81 pAsLsXsFscr
[ 1288.648152] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1288.648152] ceph: __ceph_caps_issued 00000000211593be cap 00000000651e8193 issued pFc
[ 1288.648154] ceph: 00000000211593be mode 0100644 uid.gid 0.0
[ 1288.648155] ceph: my wanted = Fc, used = Fc, dirty -
[ 1288.648156] ceph: mds wanted LsFc -> Fc
[ 1288.648157] ceph: grant: pFc -> pAsLsXsFscr
[ 1288.648158] ceph: invalidate_aliases inode 00000000211593be
[ 1288.648159] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648160] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648161] ceph: mdsc get_session 00000000c8eb37bf 17 -> 18
[ 1288.648162] ceph: mdsc con_get 00000000c8eb37bf ok (18)
[ 1288.648163] ceph: handle_reply 000000003731a03d
[ 1288.648165] ceph: __unregister_request 000000003731a03d tid 107
[ 1288.648166] ceph: got safe reply 107, mds0
[ 1288.648167] ceph: mdsc con_put 00000000c8eb37bf (17)
[ 1288.648167] ceph: mdsc put_session 00000000c8eb37bf 18 -> 17
[ 1288.648168] ceph: mdsc con_put 00000000c8eb37bf (16)
[ 1288.648169] ceph: mdsc put_session 00000000c8eb37bf 17 -> 16
[ 1288.648170] ceph: put_cap_refs 000000007dbf994a had p
[ 1288.648171] ceph: ceph_d_prune 0000000053587378
[ 1292.680151] ceph: destroy_inode 00000000211593be ino 100001ed781.fffffffffffffffe
[ 1292.680153] ceph: __ceph_remove_cap 00000000651e8193 from 00000000211593be
[ 1292.680155] ceph: put_snap_realm 1 000000008b101794 2059 -> 2058
[ 1292.680156] ceph: __cap_delay_cancel 00000000211593be
[ 1292.680157] ceph: __ceph_destroy_xattrs p= (null)
[ 1292.680158] ceph: d_release 0000000053587378
[ 1292.680159] ceph: dentry_lru_del 00000000dc920853 0000000053587378 'file3.txt'
[ 1292.680161] ceph: mdsc put_session 00000000c8eb37bf 16 -> 15
[ 1292.680162] ceph: unreserve caps ctx=000000006c55643b count=0
[ 1292.680164] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680165] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680170] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.680170] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.680175] ceph: handle_caps from mds0
[ 1292.680178] ceph: op grant ino 100001ed782.fffffffffffffffe inode 00000000dc4173bc
[ 1292.680178] ceph: mds0 seq 1202 cap seq 79
[ 1292.680180] ceph: __ceph_caps_issued 00000000dc4173bc cap 00000000655c934a issued pFc
[ 1292.680181] ceph: handle_cap_grant inode 00000000dc4173bc cap 00000000655c934a mds0 seq 79 pAsLsXsFscr
[ 1292.680182] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1292.680183] ceph: __ceph_caps_issued 00000000dc4173bc cap 00000000655c934a issued pFc
[ 1292.680184] ceph: 00000000dc4173bc mode 0100644 uid.gid 0.0
[ 1292.680186] ceph: my wanted = Fc, used = Fc, dirty -
[ 1292.680187] ceph: mds wanted LsFc -> Fc
[ 1292.680188] ceph: grant: pFc -> pAsLsXsFscr
[ 1292.680189] ceph: invalidate_aliases inode 00000000dc4173bc
[ 1292.680190] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680191] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680192] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.680192] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.680194] ceph: handle_reply 0000000075473824
[ 1292.680195] ceph: __unregister_request 0000000075473824 tid 108
[ 1292.680197] ceph: got safe reply 108, mds0
[ 1292.680198] ceph: mdsc con_put 00000000c8eb37bf (14)
[ 1292.680198] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1292.680199] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1292.680199] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1292.680201] ceph: put_cap_refs 000000007dbf994a had p
[ 1292.680202] ceph: ceph_d_prune 00000000a0a3347f
[ 1292.733086] ceph: mdsc delayed_work
[ 1292.733087] ceph: check_delayed_caps
[ 1292.733089] ceph: lookup_mds_session 00000000c8eb37bf 13
[ 1292.733089] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1292.733093] ceph: put_cap 0000000019e816a1 2096 = 2060 used + 35 resv + 1 avail
[ 1292.733094] ceph: put_cap 00000000651e8193 2096 = 2059 used + 35 resv + 2 avail
[ 1292.733095] ceph: send_cap_releases mds0 00000000f77df325
[ 1292.733096] ceph: mdsc get_session 00000000c8eb37bf 14 -> 15
[ 1292.733096] ceph: mdsc con_get 00000000c8eb37bf ok (15)
[ 1292.733097] ceph: mdsc put_session 00000000c8eb37bf 15 -> 14
[ 1296.739311] ceph: destroy_inode 00000000dc4173bc ino 100001ed782.fffffffffffffffe
[ 1296.739312] ceph: __ceph_remove_cap 00000000655c934a from 00000000dc4173bc
[ 1296.739314] ceph: put_snap_realm 1 000000008b101794 2058 -> 2057
[ 1296.739315] ceph: __cap_delay_cancel 00000000dc4173bc
[ 1296.739316] ceph: __ceph_destroy_xattrs p= (null)
[ 1296.739317] ceph: d_release 00000000a0a3347f
[ 1296.739318] ceph: dentry_lru_del 00000000b8d3da88 00000000a0a3347f 'file4.txt'
[ 1296.739319] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739320] ceph: unreserve caps ctx=000000001f1693a7 count=0
[ 1296.739322] ceph: mdsc con_put 00000000c8eb37bf (12)
[ 1296.739322] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1296.739327] ceph: mdsc get_session 00000000c8eb37bf 12 -> 13
[ 1296.739328] ceph: mdsc con_get 00000000c8eb37bf ok (13)
[ 1296.739332] ceph: handle_reply 000000004f565148
[ 1296.739332] ceph: __unregister_request 000000004f565148 tid 110
[ 1296.739333] ceph: handle_reply tid 110 result 0
[ 1296.739335] ceph: update_snap_trace deletion=0
[ 1296.739336] ceph: lookup_snap_realm 1 000000008b101794
[ 1296.739337] ceph: get_realm 000000008b101794 2057 -> 2058
[ 1296.739338] ceph: update_snap_trace 1 000000008b101794 seq 1 unchanged
[ 1296.739339] ceph: done with 1 000000008b101794, invalidated=0, 00000000d291aa69 00000000d291aa69
[ 1296.739341] ceph: fill_trace 000000004f565148 is_dentry 0 is_target 1
[ 1296.739343] ceph: get_inode on 1099513648969=100001ed749.fffffffffffffffe got 000000007dbf994a
[ 1296.739344] ceph: fill_inode 000000007dbf994a ino 100001ed749.fffffffffffffffe v 9648574 had 9648555
[ 1296.739345] ceph: get_cap ctx=000000008187b867 (35) 2096 = 2058 used + 35 resv + 3 avail
[ 1296.739346] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739347] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1296.739349] ceph: add_cap 000000007dbf994a mds0 cap 37bd31 pAsLsXsFs seq 21
[ 1296.739349] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739351] ceph: issued pAsLsXsFs, mds wanted pAsLsXsFsx, actual p, queueing
[ 1296.739352] ceph: __cap_set_timeouts 000000007dbf994a min 873 max 873
[ 1296.739352] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295964943
[ 1296.739354] ceph: add_cap inode 000000007dbf994a (100001ed749.fffffffffffffffe) cap 000000001044692e pAsLsXsFs now pAsLsXsFs seq 21 mds0
[ 1296.739355] ceph: put_cap 00000000651e8193 2096 = 2059 used + 34 resv + 3 avail
[ 1296.739356] ceph: fill_trace done err=0
[ 1296.739357] ceph: readdir_prepopulate 0 items under dn 0000000010f74030
[ 1296.739357] ceph: readdir_prepopulate done
[ 1296.739358] ceph: unreserve caps ctx=000000008187b867 count=34
[ 1296.739359] ceph: unreserve caps 2096 = 2058 used + 0 resv + 38 avail
[ 1296.739359] ceph: put_snap_realm 1 000000008b101794 2058 -> 2057
[ 1296.739363] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739363] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739365] ceph: handle_caps from mds0
[ 1296.739367] ceph: op grant ino 100001ed749.fffffffffffffffe inode 000000007dbf994a
[ 1296.739367] ceph: mds0 seq 1203 cap seq 22
[ 1296.739368] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739369] ceph: handle_cap_grant inode 000000007dbf994a cap 000000001044692e mds0 seq 22 pAsLsXsFsx
[ 1296.739370] ceph: size 0 max_size 0, i_size 40
[ 1296.739371] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFs
[ 1296.739371] ceph: 000000007dbf994a mode 040755 uid.gid 0.0
[ 1296.739373] ceph: my wanted = p, used = p, dirty -
[ 1296.739373] ceph: mds wanted pAsLsXsFsx -> p
[ 1296.739374] ceph: grant: pAsLsXsFs -> pAsLsXsFsx
[ 1296.739375] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739375] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739377] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739377] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739378] ceph: handle_caps from mds0
[ 1296.739379] ceph: op grant ino 100001ed783.fffffffffffffffe inode 000000006e7794e2
[ 1296.739380] ceph: mds0 seq 1204 cap seq 81
[ 1296.739381] ceph: __ceph_caps_issued 000000006e7794e2 cap 00000000e4d0a581 issued pFc
[ 1296.739381] ceph: handle_cap_grant inode 000000006e7794e2 cap 00000000e4d0a581 mds0 seq 81 pAsLsXsFscr
[ 1296.739382] ceph: size 107374182400 max_size 0, i_size 107374182400
[ 1296.739383] ceph: __ceph_caps_issued 000000006e7794e2 cap 00000000e4d0a581 issued pFc
[ 1296.739383] ceph: 000000006e7794e2 mode 0100644 uid.gid 0.0
[ 1296.739384] ceph: my wanted = Fc, used = Fc, dirty -
[ 1296.739384] ceph: mds wanted LsFc -> Fc
[ 1296.739385] ceph: grant: pFc -> pAsLsXsFscr
[ 1296.739386] ceph: invalidate_aliases inode 000000006e7794e2
[ 1296.739387] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739387] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739389] ceph: mdsc get_session 00000000c8eb37bf 13 -> 14
[ 1296.739389] ceph: mdsc con_get 00000000c8eb37bf ok (14)
[ 1296.739391] ceph: handle_reply 000000003cc99a51
[ 1296.739391] ceph: __unregister_request 000000003cc99a51 tid 109
[ 1296.739392] ceph: got safe reply 109, mds0
[ 1296.739393] ceph: mdsc con_put 00000000c8eb37bf (13)
[ 1296.739394] ceph: mdsc put_session 00000000c8eb37bf 14 -> 13
[ 1296.739394] ceph: mdsc con_put 00000000c8eb37bf (12)
[ 1296.739395] ceph: mdsc put_session 00000000c8eb37bf 13 -> 12
[ 1296.739396] ceph: put_cap_refs 000000007dbf994a had p
[ 1296.739397] ceph: ceph_d_prune 00000000780bc6fa
[ 1296.739419] ceph: do_request waited, got 0
[ 1296.739421] ceph: do_request 000000004f565148 done, result 0
[ 1296.739421] ceph: readdir got and parsed readdir result=0 on frag 0, end=1, complete=1, hash_order=1
[ 1296.739422] ceph: readdir frag 0 num 0 pos 2 chunk first 0
[ 1296.739424] ceph: mdsc con_put 00000000c8eb37bf (11)
[ 1296.739425] ceph: mdsc put_session 00000000c8eb37bf 12 -> 11
[ 1296.739426] ceph: mdsc con_put 00000000c8eb37bf (10)
[ 1296.739427] ceph: mdsc put_session 00000000c8eb37bf 11 -> 10
[ 1296.739429] ceph: put_cap_refs 000000007dbf994a had p
[ 1296.739430] ceph: mdsc put_session 00000000c8eb37bf 10 -> 9
[ 1296.739431] ceph: unreserve caps ctx=000000008187b867 count=0
[ 1296.739432] ceph: marking 000000007dbf994a complete and ordered
[ 1296.739433] ceph: readdir 000000007dbf994a file 00000000fa762655 done.
[ 1296.739436] ceph: readdir 000000007dbf994a file 00000000fa762655 pos 2
[ 1296.739439] ceph: release inode 000000007dbf994a file 00000000fa762655
[ 1296.739441] ceph: put_fmode 000000007dbf994a fmode 0 {0,0,0,0}
[ 1296.739442] ceph: __ceph_caps_issued 000000007dbf994a cap 000000001044692e issued pAsLsXsFsx
[ 1296.739444] ceph: check_caps 000000007dbf994a file_want - used - dirty - flushing - issued pAsLsXsFsx revoking - retain pAsLsXsFsx
[ 1296.739445] ceph: mds0 cap 000000001044692e used - issued pAsLsXsFsx implemented pAsLsXsFsx revoking -
[ 1296.739447] ceph: delaying issued pAsLsXsFsx -> pAsLsXsFsx, wanted pAsLsXsFsx -> AsLsXsFsx
[ 1296.739448] ceph: __cap_set_timeouts 000000007dbf994a min 864 max 864
[ 1296.739449] ceph: __cap_delay_requeue 000000007dbf994a flags 0 at 4295964934
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
