That is rather low, increasing the pg count should help with the data distribution.
Documentation recommends starting with (100 * (num of osds)) /(replicas) rounded up to the nearest power of two. https://ceph.com/docs/master/rados/operations/placement-groups/ On Fri, Apr 18, 2014 at 4:54 AM, Kenneth Waegeman <kenneth.waege...@ugent.be > wrote: > > ----- Message from Tyler Brekke <tyler.bre...@inktank.com> --------- > Date: Fri, 18 Apr 2014 04:37:26 -0700 > From: Tyler Brekke <tyler.bre...@inktank.com> > Subject: Re: [ceph-users] OSD distribution unequally > To: Dan Van Der Ster <daniel.vanders...@cern.ch> > Cc: Kenneth Waegeman <kenneth.waege...@ugent.be>, ceph-users < > ceph-users@lists.ceph.com> > > > > How many placement groups do you have in your pool containing the data, >> and >> what is the replication level of that pool? >> > > 400 pgs per pool, replication factor is 3 > > > >> Looks like you have too few placement groups which is causing the data >> distribution to be off. >> >> -Tyler >> >> >> On Fri, Apr 18, 2014 at 4:12 AM, Dan Van Der Ster < >> daniel.vanders...@cern.ch >> >>> wrote: >>> >> >> ceph osd reweight-by-utilization >>> >>> Is that still in 0.79? >>> >>> I'd start with reweight-by-utilization 200 and then adjust that number >>> down until you get to 120 or so. >>> >>> Cheers, Dan >>> On Apr 18, 2014 12:49 PM, Kenneth Waegeman <kenneth.waege...@ugent.be> >>> wrote: >>> Hi, >>> >>> Some osds of our cluster filled up: >>> health HEALTH_ERR 1 full osd(s); 4 near full osd(s) >>> monmap e1: 3 mons at >>> {ceph001= >>> 10.141.8.180:6789/0,ceph002=10.141.8.181:6789/0,ceph003= >>> 10.141.8.182:6789/0}, >>> election epoch 96, quorum 0,1,2 >>> ceph001,ceph002,ceph003 >>> mdsmap e93: 1/1/1 up {0=ceph001.cubone.os=up:active}, 1 up:standby >>> osdmap e1974: 42 osds: 42 up, 42 in >>> flags full >>> pgmap v286626: 1200 pgs, 3 pools, 31096 GB data, 26259 kobjects >>> 94270 GB used, 40874 GB / 131 TB avail >>> 1 active+clean+scrubbing+deep >>> 1199 active+clean >>> >>> I knew it is never really uniform, but the differences of the OSDs >>> seems very big, one OSD has 96% while another only has 48% usage, >>> which is about 1,8 TB difference: >>> /dev/sdc 3.7T 1.9T 1.8T 51% /var/lib/ceph/osd/sdc >>> /dev/sdd 3.7T 2.5T 1.2T 68% /var/lib/ceph/osd/sdd >>> /dev/sde 3.7T 2.3T 1.5T 61% /var/lib/ceph/osd/sde >>> /dev/sdf 3.7T 2.7T 975G 74% /var/lib/ceph/osd/sdf >>> /dev/sdg 3.7T 3.2T 491G 87% /var/lib/ceph/osd/sdg >>> /dev/sdh 3.7T 2.0T 1.8T 53% /var/lib/ceph/osd/sdh >>> /dev/sdi 3.7T 2.3T 1.4T 63% /var/lib/ceph/osd/sdi >>> /dev/sdj 3.7T 3.4T 303G 92% /var/lib/ceph/osd/sdj >>> /dev/sdk 3.7T 2.8T 915G 76% /var/lib/ceph/osd/sdk >>> /dev/sdl 3.7T 1.8T 2.0T 48% /var/lib/ceph/osd/sdl >>> /dev/sdm 3.7T 2.8T 917G 76% /var/lib/ceph/osd/sdm >>> /dev/sdn 3.7T 3.5T 186G 96% /var/lib/ceph/osd/sdn >>> >>> We are running 0.79 (well precisely a patched version of it with an >>> MDS fix of another thread:-) ) >>> I remember hearing something about the hashpgpool having an effect on >>> it, but I read this was already default enabled on the latest >>> versions. osd_pool_default_flag_hashpspool has indeed the value true, >>> but I don't know how to check this for a specific pool. >>> >>> Is this behaviour normal? Or what can be wrong? >>> >>> Thanks! >>> >>> Kind regards, >>> Kenneth Waegeman >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> _______________________________________________ >>> ceph-users mailing list >>> ceph-users@lists.ceph.com >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>> >>> >>> > > ----- End message from Tyler Brekke <tyler.bre...@inktank.com> ----- > > -- > > Met vriendelijke groeten, > Kenneth Waegeman > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
