On Thu, Jul 6, 2017 at 7:04 AM <bruno.cann...@stfc.ac.uk> wrote: > Hi Ceph Users, > > > > We plan to add 20 storage nodes to our existing cluster of 40 nodes, each > node has 36 x 5.458 TiB drives. We plan to add the storage such that all > new OSDs are prepared, activated and ready to take data but not until we > start slowly increasing their weightings. We also expect this not to cause > any backfilling before we adjust the weightings. > > > > When testing the deployment on our development cluster, adding a new OSD > to the host bucket with a crush weight of 5.458 and an OSD reweight of 0 > (we have set “noin”) causes the acting sets of a few pools to change, thus > triggering backfilling. Interestingly, none of the pool backfilling have > the new OSD in their acting set. > > > > This was not what we expected, so I have to ask, is what we are trying to > achieve possible and how best we should go about doing it. >
Yeah, there's an understandable but unfortunate bit where when you add a new CRUSH device/bucket to a CRUSH bucket (so, a new disk to a host, or a new host to a rack) you change the overall weight of that bucket (the host or rack). So even though the new OSD might be added with a *reweight* of zero, it has a "real" weight of 5.458 and so a little bit more data is mapped into the host/rack, even though none gets directed to the new disk until you set its reweight value up. As you note below, if you add the disks with a weight of zero that doesn't happen, so you can try doing that and weighting them up gradually. -Greg > > Commands run: > > ceph osd crush add osd.43 0 host=ceph-sn833 - causes no backfilling > > ceph osd crush add osd.44 5.458 host=ceph-sn833 - does cause backfilling > > > > For multiple hosts and OSDs, we plan to prepare a new crushmap and inject > that into the cluster. > > > > Best wishes, > > Bruno > > > > > > Bruno Canning > > LHC Data Store System Administrator > > Scientific Computing Department > > STFC Rutherford Appleton Laboratory > > Harwell Oxford > > Didcot > > OX11 0QX > > Tel. +44 ((0)1235) 446621 > > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
