Hi,Sage.

Yes, Firefly.
[root@ceph05 ~]# ceph --version
ceph version 0.80.8 (69eaad7f8308f21573c604f121956e64679a52a7)

Yes, I have seen this behavior.

[root@ceph08 ceph]# rbd info vm-160-disk-1
rbd image 'vm-160-disk-1':
        size 32768 MB in 8192 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.179faf52eb141f2
        format: 2
        features: layering
        parent: rbd/base-145-disk-1@__base__
        overlap: 32768 MB
[root@ceph08 ceph]# rbd rm vm-160-disk-1
Removing image: 100% complete...done.
[root@ceph08 ceph]# rbd info vm-160-disk-1
2015-01-28 10:39:01.595785 7f1fbea9e760 -1 librbd::ImageCtx: error finding
header: (2) No such file or directoryrbd: error opening image
vm-160-disk-1: (2) No such file or directory

[root@ceph08 ceph]# rados -p rbdcache ls | grep 179faf52eb141f2 | wc
   5944    5944  249633
[root@ceph08 ceph]# rados -p rbdcache ls | grep 179faf52eb141f2 | wc
   5857    5857  245979
[root@ceph08 ceph]# rados -p rbd ls | grep 179faf52eb141f2 | wc
   4377    4377  183819
[root@ceph08 ceph]# rados -p rbdcache ls | grep 179faf52eb141f2 | wc
   5017    5017  210699
[root@ceph08 ceph]# rados -p rbdcache ls | grep 179faf52eb141f2 | wc
   5015    5015  210615
[root@ceph08 ceph]# rados -p rbd ls | grep 179faf52eb141f2 | wc
[root@ceph08 ceph]# rados -p rcachehe ls | grep 179faf52eb141f2 | wc
   1986    1986   83412
[root@ceph08 ceph]# rados -p rbd ls | grep 179faf52eb141f2 | wc
    981     981   41202
[root@ceph08 ceph]# rados -p rbd ls | grep 179faf52eb141f2 | wc
    802     802   33684
[root@ceph08 ceph]# rados -p rbdcache ls | grep 179faf52eb141f2 | wc
   1611    1611   67662

Thank, Sage!


Tue Jan 27 2015 at 7:01:43 PM, Sage Weil <s...@newdream.net>:

> On Tue, 27 Jan 2015, Irek Fasikhov wrote:
> > Hi,All.
> > Indeed, there is a problem. Removed 1 TB of data space on a cluster is
> not
> > cleared. This feature of the behavior or a bug? And how long will it be
> > cleaned?
>
> Your subject says cache tier but I don't see it in the 'ceph df' output
> below.  The cache tiers will store 'whiteout' objects that cache object
> non-existence that could be delaying some deletion.  You can wrangle the
> cluster into flushing those with
>
>  ceph osd pool set <cachepool> cache_target_dirty_ratio .05
>
> (though you'll probably want to change it back to the default .4 later).
>
> If there's no cache tier involved, there may be another problem.  What
> version is this?  Firefly?
>
> sage
>
> >
> > Sat Sep 20 2014 at 8:19:24 AM, Mika?l Cluseau <mclus...@isi.nc>:
> >       Hi all,
> >
> >       I have weird behaviour on my firefly "test + convenience
> >       storage" cluster. It consists of 2 nodes with a light imbalance
> >       in available space:
> >
> >       # id    weight    type name    up/down    reweight
> >       -1    14.58    root default
> >       -2    8.19        host store-1
> >       1    2.73            osd.1    up    1
> >       0    2.73            osd.0    up    1
> >       5    2.73            osd.5    up    1
> >       -3    6.39        host store-2
> >       2    2.73            osd.2    up    1
> >       3    2.73            osd.3    up    1
> >       4    0.93            osd.4    up    1
> >
> >       I used to store ~8TB of rbd volumes, coming to a near-full
> >       state. There was some annoying "stuck misplaced" PGs so I began
> >       to remove 4.5TB of data; the weird thing is: the space hasn't
> >       been reclaimed on the OSDs, they keeped stuck around 84% usage.
> >       I tried to move PGs around and it happens that the space is
> >       correctly "reclaimed" if I take an OSD out, let him empty it XFS
> >       volume and then take it in again.
> >
> >       I'm currently applying this to and OSD in turn, but I though it
> >       could be worth telling about this. The current ceph df output
> >       is:
> >
> >       GLOBAL:
> >           SIZE       AVAIL     RAW USED     %RAW USED
> >           12103G     5311G     6792G        56.12
> >       POOLS:
> >           NAME                 ID     USED       %USED     OBJECTS
> >           data                 0      0          0         0
> >           metadata             1      0          0         0
> >           rbd                  2      444G       3.67      117333
> >       [...]
> >           archives-ec          14     3628G      29.98     928902
> >           archives             15     37518M     0.30      273167
> >
> >       Before "just moving data", AVAIL was around 3TB.
> >
> >       I finished the process with the OSDs on store-1, who show the
> >       following space usage now:
> >
> >       /dev/sdb1             2.8T  1.4T  1.4T  50%
> >       /var/lib/ceph/osd/ceph-0
> >       /dev/sdc1             2.8T  1.3T  1.5T  46%
> >       /var/lib/ceph/osd/ceph-1
> >       /dev/sdd1             2.8T  1.3T  1.5T  48%
> >       /var/lib/ceph/osd/ceph-5
> >
> >       I'm currently fixing OSD 2, 3 will be the last one to be fixed.
> >       The df on store-2 shows the following:
> >
> >       /dev/sdb1               2.8T  1.9T  855G  70%
> >       /var/lib/ceph/osd/ceph-2
> >       /dev/sdc1               2.8T  2.4T  417G  86%
> >       /var/lib/ceph/osd/ceph-3
> >       /dev/sdd1               932G  481G  451G  52%
> >       /var/lib/ceph/osd/ceph-4
> >
> >       OSD 2 was at 84% 3h ago, and OSD 3 was ~75%.
> >
> >       During rbd rm (that took a bit more that 3 days), ceph log was
> >       showing things like that:
> >
> >       2014-09-03 16:17:38.831640 mon.0 192.168.1.71:6789/0 417194 :
> >       [INF] pgmap v14953987: 3196 pgs: 2882 active+clean, 314
> >       active+remapped; 7647 GB data, 11067 GB used, 3828 GB / 14896 GB
> >       avail; 0 B/s rd, 6778 kB/s wr, 18 op/s; -5/5757286 objects
> >       degraded (-0.000%)
> >       [...]
> >       2014-09-05 03:09:59.895507 mon.0 192.168.1.71:6789/0 513976 :
> >       [INF] pgmap v15050766: 3196 pgs: 2882 active+clean, 314
> >       active+remapped; 6010 GB data, 11156 GB used, 3740 GB / 14896 GB
> >       avail; 0 B/s rd, 0 B/s wr, 8 op/s; -388631/5247320 objects
> >       degraded (-7.406%)
> >       [...]
> >       2014-09-06 03:56:50.008109 mon.0 192.168.1.71:6789/0 580816 :
> >       [INF] pgmap v15117604: 3196 pgs: 2882 active+clean, 314
> >       active+remapped; 4865 GB data, 11207 GB used, 3689 GB / 14896 GB
> >       avail; 0 B/s rd, 6117 kB/s wr, 22 op/s; -706519/3699415 objects
> >       degraded (-19.098%)
> >       2014-09-06 03:56:44.476903 osd.0 192.168.1.71:6805/11793 729 :
> >       [WRN] 1 slow requests, 1 included below; oldest blocked for >
> >       30.058434 secs
> >       2014-09-06 03:56:44.476909 osd.0 192.168.1.71:6805/11793 730 :
> >       [WRN] slow request 30.058434 seconds old, received at 2014-09-06
> >       03:56:14.418429: osd_op(client.19843278.0:46081
> >       rb.0.c7fd7f.238e1f29.00000000b3fa [delete] 15.b8fb7551
> >       ack+ondisk+write e38950) v4 currently waiting for blocked object
> >       2014-09-06 03:56:49.477785 osd.0 192.168.1.71:6805/11793 731 :
> >       [WRN] 2 slow requests, 1 included below; oldest blocked for >
> >       35.059315 secs
> >       [... stabilizes here:]
> >       2014-09-06 22:13:48.771531 mon.0 192.168.1.71:6789/0 632527 :
> >       [INF] pgmap v15169313: 3196 pgs: 2882 active+clean, 314
> >       active+remapped; 4139 GB data, 11215 GB used, 3681 GB / 14896 GB
> >       avail; 64 B/s rd, 64 B/s wr, 0 op/s; -883219/3420796 objects
> >       degraded (-25.819%)
> >       [...]
> >       2014-09-07 03:09:48.491325 mon.0 192.168.1.71:6789/0 633880 :
> >       [INF] pgmap v15170666: 3196 pgs: 2882 active+clean, 314
> >       active+remapped; 4139 GB data, 11215 GB used, 3681 GB / 14896 GB
> >       avail; 18727 B/s wr, 2 op/s; -883219/3420796 objects degraded
> >       (-25.819%)
> >
> >       And now, during data movement I described before:
> >
> >       2014-09-20 15:16:13.394694 mon.0 [INF] pgmap v15344707: 3196
> >       pgs: 2132 active+clean, 432 active+remapped+wait_backfill, 621
> >       active+remapped, 11 active+remapped+backfilling; 4139 GB data,
> >       6831 GB used, 5271 GB / 12103 GB avail; 379097/3792969 objects
> >       degraded (9.995%)
> >
> >       If some ceph developer wants me to do something or to provide
> >       some data, please say so quickly, I will probably process OSD 3
> >       in ~16-20h.
> >       (of course, I'd prefer not loose the data btw :-))
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> >
> >
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to