----- Message from Tyler Brekke <tyler.bre...@inktank.com> ---------
Date: Fri, 18 Apr 2014 04:37:26 -0700
From: Tyler Brekke <tyler.bre...@inktank.com>
Subject: Re: [ceph-users] OSD distribution unequally
To: Dan Van Der Ster <daniel.vanders...@cern.ch>
Cc: Kenneth Waegeman <kenneth.waege...@ugent.be>, ceph-users
<ceph-users@lists.ceph.com>
How many placement groups do you have in your pool containing the data, and
what is the replication level of that pool?
400 pgs per pool, replication factor is 3
Looks like you have too few placement groups which is causing the data
distribution to be off.
-Tyler
On Fri, Apr 18, 2014 at 4:12 AM, Dan Van Der Ster <daniel.vanders...@cern.ch
wrote:
ceph osd reweight-by-utilization
Is that still in 0.79?
I'd start with reweight-by-utilization 200 and then adjust that number
down until you get to 120 or so.
Cheers, Dan
On Apr 18, 2014 12:49 PM, Kenneth Waegeman <kenneth.waege...@ugent.be>
wrote:
Hi,
Some osds of our cluster filled up:
health HEALTH_ERR 1 full osd(s); 4 near full osd(s)
monmap e1: 3 mons at
{ceph001=
10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,ceph003=10.141.8.182:6789/0},
election epoch 96, quorum 0,1,2
ceph001,ceph002,ceph003
mdsmap e93: 1/1/1 up {0=ceph001.cubone.os=up:active}, 1 up:standby
osdmap e1974: 42 osds: 42 up, 42 in
flags full
pgmap v286626: 1200 pgs, 3 pools, 31096 GB data, 26259 kobjects
94270 GB used, 40874 GB / 131 TB avail
1 active+clean+scrubbing+deep
1199 active+clean
I knew it is never really uniform, but the differences of the OSDs
seems very big, one OSD has 96% while another only has 48% usage,
which is about 1,8 TB difference:
/dev/sdc 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/sdc
/dev/sdd 3.7T 2.5T 1.2T 68% /var/lib/ceph/osd/sdd
/dev/sde 3.7T 2.3T 1.5T 61% /var/lib/ceph/osd/sde
/dev/sdf 3.7T 2.7T 975G 74% /var/lib/ceph/osd/sdf
/dev/sdg 3.7T 3.2T 491G 87% /var/lib/ceph/osd/sdg
/dev/sdh 3.7T 2.0T 1.8T 53% /var/lib/ceph/osd/sdh
/dev/sdi 3.7T 2.3T 1.4T 63% /var/lib/ceph/osd/sdi
/dev/sdj 3.7T 3.4T 303G 92% /var/lib/ceph/osd/sdj
/dev/sdk 3.7T 2.8T 915G 76% /var/lib/ceph/osd/sdk
/dev/sdl 3.7T 1.8T 2.0T 48% /var/lib/ceph/osd/sdl
/dev/sdm 3.7T 2.8T 917G 76% /var/lib/ceph/osd/sdm
/dev/sdn 3.7T 3.5T 186G 96% /var/lib/ceph/osd/sdn
We are running 0.79 (well precisely a patched version of it with an
MDS fix of another thread:-) )
I remember hearing something about the hashpgpool having an effect on
it, but I read this was already default enabled on the latest
versions. osd_pool_default_flag_hashpspool has indeed the value true,
but I don't know how to check this for a specific pool.
Is this behaviour normal? Or what can be wrong?
Thanks!
Kind regards,
Kenneth Waegeman
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
----- End message from Tyler Brekke <tyler.bre...@inktank.com> -----
--
Met vriendelijke groeten,
Kenneth Waegeman
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
