On Fri, Sep 13, 2013 at 3:09 PM, Jens-Christian Fischer <
jens-christian.fisc...@switch.ch> wrote:
> Hi all
>
> we have started to use CephFS as the backing storage for OpenStack VM
> instance images. We have one pool (backed by SSDs) in our CephCluster that
> is exposed via CephFS to the different physical hosts on our machine.
>
> The problem we see, is that the different hosts have different views on
> the filesystem (i.e. they see different amount of files). See an example
> listing at the end - this lists the virtual disks that each server sees
> (h0, h1 etc are the name of the servers)
>
> All servers mount the same filesystem. Needless to say, that we are a bit
> worried…
>
The bug was introduced in 3.10 kernel, will be fixed in 3.12 kernel by
commit 590fb51f1c (vfs: call d_op->d_prune() before unhashing dentry). Sage
may backport the fix to 3.11 and 3.10 kernel soon. please use ceph-fuse at
present.
Regards
Yan, Zheng
>
> Any ideas?
> cheers
> jc
>
> root@ineri ~$ ndo all_nodes grep instances /etc/fstab
> h0
> [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances
> /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2
> h1
> [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances
> /mnt/instances ceph name=admin,secret=...g==roo,noatime 0 2
> h3
> [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances
> /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2
> h5
> [2001:620:0:6::106]:6789,[2001:620:0:6::10e]:6789,[2001:620:0:6::10c]:6789:/instances
> /mnt/instances ceph name=admin,secret=...==roo,noatime 0 2
>
> We have upgraded theses servers to the 3.10.10 kernel
>
> root@ineri ~$ ndo all_nodes uname -a
> h0 Linux h0 3.10.10-031010-generic #201308291422 SMP Thu Aug 29
> 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> h1 Linux h1 3.10.10-031010-generic #201308291422 SMP Thu Aug 29
> 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> h3 Linux h3 3.10.10-031010-generic #201308291422 SMP Thu Aug 29
> 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> h5 Linux h5 3.10.10-031010-generic #201308291422 SMP Thu Aug 29
> 18:24:39 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> and are running ceph 61.7
>
> root@ineri ~$ ndo all_nodes ceph --version
> h0 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> h1 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> h2 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> h3 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> h4 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
> h5 ceph version 0.61.7 (8f010aff684e820ecc837c25ac77c7a05d7191ff)
>
> mds is running on two other nodes, these are on 61.8
>
> root@ineri ~$ ceph --version
> ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
> root@myoko:~# ceph --version
> ceph version 0.61.8 (a6fdcca3bddbc9f177e4e2bf0d9cdd85006b028b)
>
>
>
> --- cut ---
> root@ineri ~$ ndo all_nodes ls -l /mnt/instances/instance-*/disk
> h0 {
> -rw-r--r-- 1 root root 4050518016 Aug 30 11:23
> /mnt/instances/instance-00000053/disk
> -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54
> /mnt/instances/instance-00000059/disk
> -rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970
> /mnt/instances/instance-0000005e/disk
> -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54
> /mnt/instances/instance-000000ec/disk
> -rw-r--r-- 1 root root 1245773824 Sep 13 08:54
> /mnt/instances/instance-00000133/disk
> -rw-r--r-- 1 root root 414580736 Sep 13 08:54
> /mnt/instances/instance-0000013d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54
> /mnt/instances/instance-0000016a/disk
> -rw-r--r-- 1 root root 4248043520 Sep 13 08:54
> /mnt/instances/instance-0000074a/disk
> -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54
> /mnt/instances/instance-0000074f/disk
> -rw-r--r-- 1 root root 1503854592 Sep 5 18:45
> /mnt/instances/instance-0000075f/disk
> -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54
> /mnt/instances/instance-0000076b/disk
> -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54
> /mnt/instances/instance-0000077d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54
> /mnt/instances/instance-00000792/disk}
>
> h1 {
> -rw-r--r-- 1 root root 4050518016 Aug 30 11:23
> /mnt/instances/instance-00000053/disk
> -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54
> /mnt/instances/instance-00000059/disk
> -rw-r--r-- 1 libvirt-qemu kvm 0 Jan 1 1970
> /mnt/instances/instance-0000005e/disk
> -rw-r--r-- 1 libvirt-qemu kvm 197120 Apr 8 14:25
> /mnt/instances/instance-0000006e/disk
> -rw-r--r-- 1 root root 1545994240 Sep 5 23:25
> /mnt/instances/instance-00000070/disk
> -rw-r--r-- 1 root root 603979776 Sep 5 23:22
> /mnt/instances/instance-00000073/disk
> -rw-r--r-- 1 root root 429654016 Jun 18 09:16
> /mnt/instances/instance-000000a4/disk
> -rw-r--r-- 1 root root 1985019904 Aug 29 14:41
> /mnt/instances/instance-000000a7/disk
> -rw-r--r-- 1 nova nova 197120 May 22 23:28
> /mnt/instances/instance-000000c7/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 14:11
> /mnt/instances/instance-000000c8/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 18:33
> /mnt/instances/instance-000000c9/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 18:35
> /mnt/instances/instance-000000ca/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 18:46
> /mnt/instances/instance-000000cb/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 19:14
> /mnt/instances/instance-000000cc/disk
> -rw-r--r-- 1 nova nova 197120 May 27 15:19
> /mnt/instances/instance-000000cd/disk
> -rw-r--r-- 1 nova nova 197120 May 27 15:29
> /mnt/instances/instance-000000ce/disk
> -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54
> /mnt/instances/instance-000000ec/disk
> -rw-r--r-- 1 root root 1245773824 Sep 13 08:54
> /mnt/instances/instance-00000133/disk
> -rw-r--r-- 1 root root 414580736 Sep 13 08:54
> /mnt/instances/instance-0000013d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54
> /mnt/instances/instance-0000016a/disk
> -rw-r--r-- 1 nova nova 221380608 Sep 6 16:06
> /mnt/instances/instance-000003ee/disk
> -rw-r--r-- 1 root root 4248043520 Sep 13 08:54
> /mnt/instances/instance-0000074a/disk
> -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54
> /mnt/instances/instance-0000074f/disk
> -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54
> /mnt/instances/instance-0000076b/disk
> -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54
> /mnt/instances/instance-0000077d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54
> /mnt/instances/instance-00000792/disk}
>
> h3 {
> -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54
> /mnt/instances/instance-00000059/disk
> -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54
> /mnt/instances/instance-000000ec/disk
> -rw-r--r-- 1 root root 1245773824 Sep 13 08:54
> /mnt/instances/instance-00000133/disk
> -rw-r--r-- 1 root root 414580736 Sep 13 08:54
> /mnt/instances/instance-0000013d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54
> /mnt/instances/instance-0000016a/disk
> -rw-r--r-- 1 root root 2193424896 Sep 10 16:47
> /mnt/instances/instance-00000193/disk
> -rw-r--r-- 1 root root 4248043520 Sep 13 08:54
> /mnt/instances/instance-0000074a/disk
> -rw-r--r-- 1 libvirt-qemu kvm 383909888 Sep 13 02:43
> /mnt/instances/instance-0000074e/disk
> -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54
> /mnt/instances/instance-0000074f/disk
> -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54
> /mnt/instances/instance-0000076b/disk
> -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54
> /mnt/instances/instance-0000077d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54
> /mnt/instances/instance-00000792/disk}
>
> h5 {
> -rw-r--r-- 1 libvirt-qemu kvm 1030225920 Sep 13 08:54
> /mnt/instances/instance-00000059/disk
> -rw-r--r-- 1 nova nova 197120 May 22 23:28
> /mnt/instances/instance-000000c7/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 14:11
> /mnt/instances/instance-000000c8/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 18:33
> /mnt/instances/instance-000000c9/disk
> -rw-r--r-- 1 nova nova 1048576 May 24 18:35
> /mnt/instances/instance-000000ca/disk
> -rw-r--r-- 1 nova nova 197120 May 27 15:19
> /mnt/instances/instance-000000cd/disk
> -rw-r--r-- 1 nova nova 197120 May 27 15:29
> /mnt/instances/instance-000000ce/disk
> -rw-r--r-- 1 nova nova 197120 May 27 15:38
> /mnt/instances/instance-000000cf/disk
> -rw-r--r-- 1 nova nova 1048576 May 27 16:36
> /mnt/instances/instance-000000d1/disk
> -rw-r--r-- 1 nova nova 1048576 May 27 16:36
> /mnt/instances/instance-000000d2/disk
> -rw-r--r-- 1 nova nova 1048576 May 27 16:36
> /mnt/instances/instance-000000d5/disk
> -rw-r--r-- 1 libvirt-qemu 998 3594715136 Sep 13 08:54
> /mnt/instances/instance-000000ec/disk
> -rw-r--r-- 1 root root 1245773824 Sep 13 08:54
> /mnt/instances/instance-00000133/disk
> -rw-r--r-- 1 root root 414580736 Sep 13 08:54
> /mnt/instances/instance-0000013d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 569311232 Sep 13 08:54
> /mnt/instances/instance-0000016a/disk
> -rw-r--r-- 1 nova nova 221380608 Sep 6 16:06
> /mnt/instances/instance-000003ee/disk
> -rw-r--r-- 1 root root 4248043520 Sep 13 08:54
> /mnt/instances/instance-0000074a/disk
> -rw-r--r-- 1 libvirt-qemu kvm 508624896 Sep 13 08:54
> /mnt/instances/instance-0000074f/disk
> -rw-r--r-- 1 libvirt-qemu kvm 5176492032 Sep 13 08:54
> /mnt/instances/instance-0000076b/disk
> -rw-r--r-- 1 libvirt-qemu kvm 336789504 Sep 13 08:54
> /mnt/instances/instance-0000077d/disk
> -rw-r--r-- 1 libvirt-qemu kvm 219152384 Sep 13 08:54
> /mnt/instances/instance-00000792/disk}
>
>
> --- cut ---
>
> --
> SWITCH
> Jens-Christian Fischer, Peta Solutions
> Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
> phone +41 44 268 15 15, direct +41 44 268 15 71
> jens-christian.fisc...@switch.ch
> http://www.switch.ch
>
> http://www.switch.ch/socialmedia
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
