Glen, Correction...looked at the wrong column for weights, my bad...
I was looking at the wrong column for weight. You have varying weights, but the process is still the same. Balance your buckets (hosts) in your crush map, and balance your osds in each bucket (host). On Sat, Jul 21, 2018 at 9:14 AM, Shawn Iverson <ivers...@rushville.k12.in.us > wrote: > Glen, > > It appears you have 447G, 931G, and 558G disks in your cluster, all with a > weight of 1.0. This means that although the new disks are bigger, they are > not going to be utilized by pgs any more than any other disk. > > I would suggest reweighting your other disks (they are smaller), so that > you balance your cluster. You should do this gradually over time, > preferably during off-peak times, when remapping will not affect operations. > > I do a little math, first by taking total cluster capacity and dividing it > by total capacity of each bucket. I then do the same thing in each bucket, > until everything is proportioned appropriately down to the osds. > > On Fri, Jul 20, 2018 at 8:43 PM, Glen Baars <g...@onsitecomputers.com.au> > wrote: > >> Hello Ceph Users, >> >> >> >> We have added more ssd storage to our ceph cluster last night. We added 4 >> x 1TB drives and the available space went from 1.6TB to 0.6TB ( in `ceph >> df` for the SSD pool ). >> >> >> >> I would assume that the weight needs to be changed but I didn’t think I >> would need to? Should I change them to 0.75 from 0.9 and hopefully it will >> rebalance correctly? >> >> >> >> #ceph osd tree | grep -v hdd >> >> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF >> >> -1 534.60309 root default >> >> -19 62.90637 host NAS-AUBUN-RK2-CEPH06 >> >> 115 ssd 0.43660 osd.115 up 1.00000 1.00000 >> >> 116 ssd 0.43660 osd.116 up 1.00000 1.00000 >> >> 117 ssd 0.43660 osd.117 up 1.00000 1.00000 >> >> 118 ssd 0.43660 osd.118 up 1.00000 1.00000 >> >> -22 105.51169 host NAS-AUBUN-RK2-CEPH07 >> >> 138 ssd 0.90970 osd.138 up 1.00000 1.00000 >> Added >> >> 139 ssd 0.90970 osd.139 up 1.00000 1.00000 >> Added >> >> -25 105.51169 host NAS-AUBUN-RK2-CEPH08 >> >> 140 ssd 0.90970 osd.140 up 1.00000 1.00000 >> Added >> >> 141 ssd 0.90970 osd.141 up 1.00000 1.00000 >> Added >> >> -3 56.32617 host NAS-AUBUN-RK3-CEPH01 >> >> 60 ssd 0.43660 osd.60 up 1.00000 1.00000 >> >> 61 ssd 0.43660 osd.61 up 1.00000 1.00000 >> >> 62 ssd 0.43660 osd.62 up 1.00000 1.00000 >> >> 63 ssd 0.43660 osd.63 up 1.00000 1.00000 >> >> -5 56.32617 host NAS-AUBUN-RK3-CEPH02 >> >> 64 ssd 0.43660 osd.64 up 1.00000 1.00000 >> >> 65 ssd 0.43660 osd.65 up 1.00000 1.00000 >> >> 66 ssd 0.43660 osd.66 up 1.00000 1.00000 >> >> 67 ssd 0.43660 osd.67 up 1.00000 1.00000 >> >> -7 56.32617 host NAS-AUBUN-RK3-CEPH03 >> >> 68 ssd 0.43660 osd.68 up 1.00000 1.00000 >> >> 69 ssd 0.43660 osd.69 up 1.00000 1.00000 >> >> 70 ssd 0.43660 osd.70 up 1.00000 1.00000 >> >> 71 ssd 0.43660 osd.71 up 1.00000 1.00000 >> >> -13 45.84741 host NAS-AUBUN-RK3-CEPH04 >> >> 72 ssd 0.54579 osd.72 up 1.00000 1.00000 >> >> 73 ssd 0.54579 osd.73 up 1.00000 1.00000 >> >> 76 ssd 0.54579 osd.76 up 1.00000 1.00000 >> >> 77 ssd 0.54579 osd.77 up 1.00000 1.00000 >> >> -16 45.84741 host NAS-AUBUN-RK3-CEPH05 >> >> 74 ssd 0.54579 osd.74 up 1.00000 1.00000 >> >> 75 ssd 0.54579 osd.75 up 1.00000 1.00000 >> >> 78 ssd 0.54579 osd.78 up 1.00000 1.00000 >> >> 79 ssd 0.54579 osd.79 up 1.00000 1.00000 >> >> >> >> # ceph osd df | grep -v hdd >> >> ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS >> >> 115 ssd 0.43660 1.00000 447G 250G 196G 56.00 1.72 103 >> >> 116 ssd 0.43660 1.00000 447G 191G 255G 42.89 1.32 84 >> >> 117 ssd 0.43660 1.00000 447G 213G 233G 47.79 1.47 92 >> >> 118 ssd 0.43660 1.00000 447G 208G 238G 46.61 1.43 85 >> >> 138 ssd 0.90970 1.00000 931G 820G 111G 88.08 2.71 216 Added >> >> 139 ssd 0.90970 1.00000 931G 771G 159G 82.85 2.55 207 Added >> >> 140 ssd 0.90970 1.00000 931G 709G 222G 76.12 2.34 197 Added >> >> 141 ssd 0.90970 1.00000 931G 664G 267G 71.31 2.19 184 Added >> >> 60 ssd 0.43660 1.00000 447G 275G 171G 61.62 1.89 100 >> >> 61 ssd 0.43660 1.00000 447G 237G 209G 53.04 1.63 90 >> >> 62 ssd 0.43660 1.00000 447G 275G 171G 61.58 1.89 95 >> >> 63 ssd 0.43660 1.00000 447G 260G 187G 58.15 1.79 97 >> >> 64 ssd 0.43660 1.00000 447G 232G 214G 52.08 1.60 83 >> >> 65 ssd 0.43660 1.00000 447G 207G 239G 46.36 1.42 75 >> >> 66 ssd 0.43660 1.00000 447G 217G 230G 48.54 1.49 84 >> >> 67 ssd 0.43660 1.00000 447G 252G 195G 56.36 1.73 92 >> >> 68 ssd 0.43660 1.00000 447G 248G 198G 55.56 1.71 94 >> >> 69 ssd 0.43660 1.00000 447G 229G 217G 51.25 1.57 84 >> >> 70 ssd 0.43660 1.00000 447G 259G 187G 58.01 1.78 87 >> >> 71 ssd 0.43660 1.00000 447G 267G 179G 59.83 1.84 97 >> >> 72 ssd 0.54579 1.00000 558G 217G 341G 38.96 1.20 100 >> >> 73 ssd 0.54579 1.00000 558G 283G 275G 50.75 1.56 121 >> >> 76 ssd 0.54579 1.00000 558G 286G 272G 51.33 1.58 129 >> >> 77 ssd 0.54579 1.00000 558G 246G 312G 44.07 1.35 104 >> >> 74 ssd 0.54579 1.00000 558G 273G 285G 48.91 1.50 122 >> >> 75 ssd 0.54579 1.00000 558G 281G 276G 50.45 1.55 114 >> >> 78 ssd 0.54579 1.00000 558G 289G 269G 51.80 1.59 133 >> >> 79 ssd 0.54579 1.00000 558G 276G 282G 49.39 1.52 119 >> >> Kind regards, >> >> *Glen Baars* >> >> BackOnline Manager >> >> >> This e-mail is intended solely for the benefit of the addressee(s) and >> any other named recipient. It is confidential and may contain legally >> privileged or confidential information. If you are not the recipient, any >> use, distribution, disclosure or copying of this e-mail is prohibited. The >> confidentiality and legal privilege attached to this communication is not >> waived or lost by reason of the mistaken transmission or delivery to you. >> If you have received this e-mail in error, please notify us immediately. >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> > > > -- > Shawn Iverson, CETL > Director of Technology > Rush County Schools > 765-932-3901 x1171 > ivers...@rushville.k12.in.us > > > -- Shawn Iverson, CETL Director of Technology Rush County Schools 765-932-3901 x1171 ivers...@rushville.k12.in.us
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
