2016-12-21 19:51 GMT+01:00 Ilya Dryomov <idryo...@gmail.com>: > On Wed, Dec 21, 2016 at 6:58 PM, Stéphane Klein > <cont...@stephane-klein.info> wrote: > >> > > 2016-12-21 18:47 GMT+01:00 Ilya Dryomov <idryo...@gmail.com>: > >> > >> On Wed, Dec 21, 2016 at 5:50 PM, Stéphane Klein > >> <cont...@stephane-klein.info> wrote: > >> > I have configured: > >> > > >> > ``` > >> > ceph osd crush tunables firefly > >> > ``` > >> > >> If it gets to rm, then it's probably not tunables. Are you running > >> these commands by hand? > > > > > > Yes, I have executed this command on my mon-1 host > > > >> > >> > >> Anything in dmesg? > > > > > > > > This: > > > > ``` > > [ 614.278589] SELinux: initialized (dev tmpfs, type tmpfs), uses > transition > > SIDs > > [ 910.797793] SELinux: initialized (dev tmpfs, type tmpfs), uses > transition > > SIDs > > [ 1126.251705] SELinux: initialized (dev tmpfs, type tmpfs), uses > transition > > SIDs > > [ 1214.030659] Key type dns_resolver registered > > [ 1214.042308] Key type ceph registered > > [ 1214.043852] libceph: loaded (mon/osd proto 15/24) > > [ 1214.045944] rbd: loaded (major 252) > > [ 1214.053449] libceph: client4200 fsid 7ecb6ebd-2e7a-44c3-bf0d- > ff8d193e03ac > > [ 1214.056406] libceph: mon0 172.28.128.2:6789 session established > > [ 1214.066596] rbd0: unknown partition table > > [ 1214.066875] rbd: rbd0: added with size 0x19000000 > > [ 1219.120342] EXT4-fs (rbd0): mounted filesystem with ordered data mode. > > Opts: (null) > > [ 1219.120754] SELinux: initialized (dev rbd0, type ext4), uses xattr > > ``` > > > > If I reboot my client host, and remount this disk then I can delete the > > folder with "rm -rf" with success. > > I'm not following - you are running 7 VMs - 3 mon VMs, 3 osd VMs and > a "ceph-client" VM. If ceph-client is debian the test case works
Yes > if it's something RHEL-based it doesn't, correct? > Yes > > Are they all on the same bare metal host? Which of these are you > rebooting? > this is this host in Vagrantfile https://github.com/harobed/poc-ceph-ansible/blob/master/vagrant-3mons-3osd/Vagrantfile#L61 > > Try dumping /sys/kernel/debug/ceph/<fsid.clientid>/osdc on the > ceph-client VM when rm hangs. osdc file is empty: # cat /sys/kernel/debug/ceph/7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac.client4200/osdc # cat /sys/kernel/debug/ceph/7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac.client4200/client_options name=admin,secret=<hidden> # cat /sys/kernel/debug/ceph/7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac.client4200/monc have osdmap 19 # cat /sys/kernel/debug/ceph/7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac.client4200/monmap epoch 1 mon0 172.28.128.2:6789 mon1 172.28.128.3:6789 mon2 172.28.128.4:6789 # cat /sys/kernel/debug/ceph/7ecb6ebd-2e7a-44c3-bf0d-ff8d193e03ac.client4200/osdmap epoch 19 flags pool 0 pg_num 64 (63) read_tier -1 write_tier -1 osd0 172.28.128.6:6800 100% (exists, up) 100% osd1 172.28.128.5:6800 100% (exists, up) 100% osd2 172.28.128.7:6800 100% (exists, up) 100% Version informations: # cat /etc/centos-release CentOS Linux release 7.2.1511 (Core) # cat /etc/centos-release-upstream Derived from Red Hat Enterprise Linux 7.2 (Source) # uname --all Linux ceph-client-1 3.10.0-327.36.1.el7.x86_64 #1 SMP Sun Sep 18 13:04:29 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux # ceph --version ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) Thanks for your help. Best regards, Stéphane
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
