How many placement groups do you have in your pool containing the data, and
what is the replication level of that pool?
Looks like you have too few placement groups which is causing the data
distribution to be off.
-Tyler
On Fri, Apr 18, 2014 at 4:12 AM, Dan Van Der Ster <daniel.vanders...@cern.ch
> wrote:
> ceph osd reweight-by-utilization
>
> Is that still in 0.79?
>
> I'd start with reweight-by-utilization 200 and then adjust that number
> down until you get to 120 or so.
>
> Cheers, Dan
> On Apr 18, 2014 12:49 PM, Kenneth Waegeman <kenneth.waege...@ugent.be>
> wrote:
> Hi,
>
> Some osds of our cluster filled up:
> health HEALTH_ERR 1 full osd(s); 4 near full osd(s)
> monmap e1: 3 mons at
> {ceph001=
> 10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,ceph003=10.141.8.182:6789/0},
> election epoch 96, quorum 0,1,2
> ceph001,ceph002,ceph003
> mdsmap e93: 1/1/1 up {0=ceph001.cubone.os=up:active}, 1 up:standby
> osdmap e1974: 42 osds: 42 up, 42 in
> flags full
> pgmap v286626: 1200 pgs, 3 pools, 31096 GB data, 26259 kobjects
> 94270 GB used, 40874 GB / 131 TB avail
> 1 active+clean+scrubbing+deep
> 1199 active+clean
>
> I knew it is never really uniform, but the differences of the OSDs
> seems very big, one OSD has 96% while another only has 48% usage,
> which is about 1,8 TB difference:
> /dev/sdc 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/sdc
> /dev/sdd 3.7T 2.5T 1.2T 68% /var/lib/ceph/osd/sdd
> /dev/sde 3.7T 2.3T 1.5T 61% /var/lib/ceph/osd/sde
> /dev/sdf 3.7T 2.7T 975G 74% /var/lib/ceph/osd/sdf
> /dev/sdg 3.7T 3.2T 491G 87% /var/lib/ceph/osd/sdg
> /dev/sdh 3.7T 2.0T 1.8T 53% /var/lib/ceph/osd/sdh
> /dev/sdi 3.7T 2.3T 1.4T 63% /var/lib/ceph/osd/sdi
> /dev/sdj 3.7T 3.4T 303G 92% /var/lib/ceph/osd/sdj
> /dev/sdk 3.7T 2.8T 915G 76% /var/lib/ceph/osd/sdk
> /dev/sdl 3.7T 1.8T 2.0T 48% /var/lib/ceph/osd/sdl
> /dev/sdm 3.7T 2.8T 917G 76% /var/lib/ceph/osd/sdm
> /dev/sdn 3.7T 3.5T 186G 96% /var/lib/ceph/osd/sdn
>
> We are running 0.79 (well precisely a patched version of it with an
> MDS fix of another thread:-) )
> I remember hearing something about the hashpgpool having an effect on
> it, but I read this was already default enabled on the latest
> versions. osd_pool_default_flag_hashpspool has indeed the value true,
> but I don't know how to check this for a specific pool.
>
> Is this behaviour normal? Or what can be wrong?
>
> Thanks!
>
> Kind regards,
> Kenneth Waegeman
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
