Hi Ali, > Message: 1 > Date: Sat, 05 Oct 2013 09:22:22 +0300 > From: Linux Chips <linux.ch...@gmail.com> > To: ceph-users@lists.ceph.com > Subject: [ceph-users] ceph uses too much disk space!! > Message-ID: <524fb01e.3000...@gmail.com> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Hi every one; > we have a small testing cluster, one node with 4 OSDs of 3TB each. i > created one RBD image of 4TB. now the cluster is nearly full: <SNIP> > /dev/sda 2.8T 2.1T 566G 79% /var/lib/ceph/osd/ceph-0 > /dev/sdb 2.8T 2.4T 316G 89% /var/lib/ceph/osd/ceph-1 > /dev/sdc 2.8T 2.2T 457G 84% /var/lib/ceph/osd/ceph-2 > /dev/sdd 2.8T 2.2T 447G 84% /var/lib/ceph/osd/ceph-3 > > # ceph osd pool get rbd min_size > min_size: 1 > > # ceph osd pool get rbd size > size: 2 > > > 4 disk at 3TB should give me 12TB, and 4TBx2 should be 8TB. that is 66% > not 80% as the ceph df shows (%RAW). > where is this space is leaking? how can i fix it? > or is this normal behavior and this is due to overhead?
I'm not sure what overhead there might be from Ceph's metadata, but I think you might be basing your calculations on bad assumptions to begin with: 1) define your 3TB OSD drive size properly (probably 3000GB or slightly less) 2) you need to account for OSD filesystem overhead, e.g., format a drive and take a look at usage (probably <95% of raw capacity) Indeed, if you look at your df output you've got ~2.8TB of total capacity per drive, and if you observed that prior to writing any data you'd see some already in-use by the filesystem. And df is showing just over 2TB per drive used, which makes sense given you've created a 4TiB rbd in a storage pool with a replication factor of 2 (i.e., 2 copies of your rbd are stored across the osds). Also, keep in mind that Ceph is breaking up your rbd into chunks on the host filesystem/s and probably storing metadata (in extended attributes) for every chunk. -- Cheers, ~Blairo
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
