OK thanks. Indeed "prepared 0/10 changes" means it thinks things are balanced. Could you again share the full ceph osd df tree?
On Mon, Mar 15, 2021 at 2:54 PM Boris Behrens <b...@kervyn.de> wrote: > > Hi Dan, > > I've set the autoscaler to warn, but it actually does not warn for now. So > not touching it for now. > > this is what the log says in minute intervals: > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/active > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/sleep_interval > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/begin_time > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/end_time > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/begin_weekday > 2021-03-15 13:51:00.970 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/end_weekday > 2021-03-15 13:51:00.971 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/pool_ids > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Optimize plan > auto_2021-03-15_13:51:00 > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/mode > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] Mode upmap, max > misplaced 0.050000 > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] do_upmap > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/upmap_max_iterations > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr get_config get_config key: > mgr/balancer/upmap_max_deviation > 2021-03-15 13:51:01.203 7f307d5fd700 4 mgr[balancer] pools > ['eu-msg-1.rgw.data.root', 'eu-msg-1.rgw.buckets.non-ec', > 'eu-central-1.rgw.users.keys', 'eu-central-1.rgw.gc', > 'eu-central-1.rgw.buckets.data', 'eu-central-1.rgw.users.email', > 'eu-msg-1.rgw.gc', 'eu-central-1.rgw.usage', 'eu-msg-1.rgw.users.keys', > 'eu-central-1.rgw.buckets.index', 'rbd', 'eu-msg-1.rgw.log', > 'whitespace-again-2021-03-10_2', 'eu-msg-1.rgw.buckets.index', > 'eu-msg-1.rgw.meta', 'eu-central-1.rgw.log', 'default.rgw.gc', > 'eu-central-1.rgw.buckets.non-ec', 'eu-msg-1.rgw.usage', > 'whitespace-again-2021-03-10', 'fra-1.rgw.meta', > 'eu-central-1.rgw.users.uid', 'eu-msg-1.rgw.users.email', > 'fra-1.rgw.control', 'eu-msg-1.rgw.users.uid', 'eu-msg-1.rgw.control', > '.rgw.root', 'eu-msg-1.rgw.buckets.data', 'default.rgw.control', > 'fra-1.rgw.log', 'default.rgw.data.root', 'whitespace-again-2021-03-10_3', > 'default.rgw.log', 'eu-central-1.rgw.meta', 'eu-central-1.rgw.data.root', > 'default.rgw.users.uid', 'eu-central-1.rgw.control'] > 2021-03-15 13:51:01.224 7f307d5fd700 4 mgr[balancer] prepared 0/10 changes > > Am Mo., 15. März 2021 um 14:15 Uhr schrieb Dan van der Ster > <d...@vanderster.com>: >> >> I suggest to just disable the autoscaler until your balancing is understood. >> >> What does your active mgr log say (with debug_mgr 4/5), grep balancer >> /var/log/ceph/ceph-mgr.*.log >> >> -- Dan >> >> On Mon, Mar 15, 2021 at 1:47 PM Boris Behrens <b...@kervyn.de> wrote: >> > >> > Hi, >> > this unfortunally did not solve my problem. I still have some OSDs that >> > fill up to 85% >> > >> > According to the logging, the autoscaler might want to add more PGs to one >> > Bucken and reduce almost all other buckets to 32. >> > 2021-03-15 12:19:58.825 7f307f601700 4 mgr[pg_autoscaler] Pool >> > 'eu-central-1.rgw.buckets.data' root_id -1 using 0.705080476146 of space, >> > bias 1.0, pg target 1974.22533321 quantized to 2048 (current 1024) >> > >> > Why the balancing does not happen is still nebulous to me. >> > >> > >> > >> > Am Sa., 13. März 2021 um 16:37 Uhr schrieb Dan van der Ster >> > <d...@vanderster.com>: >> >> >> >> OK >> >> Btw, you might need to fail to a new mgr... I'm not sure if the current >> >> active will read that new config. >> >> >> >> .. dan >> >> >> >> >> >> On Sat, Mar 13, 2021, 4:36 PM Boris Behrens <b...@kervyn.de> wrote: >> >>> >> >>> Hi, >> >>> >> >>> ok thanks. I just changed the value and rewighted everything back to 1. >> >>> Now I let it sync the weekend and check how it will be on monday. >> >>> We tried to have the systems total storage balanced as possible. New >> >>> systems will be with 8TB disks but for the exiting ones we added 16TB to >> >>> offset the 4TB disks and we needed a lot of storage fast, because of a >> >>> DC move. If you have any recommendations I would be happy to hear them. >> >>> >> >>> Cheers >> >>> Boris >> >>> >> >>> Am Sa., 13. März 2021 um 16:20 Uhr schrieb Dan van der Ster >> >>> <d...@vanderster.com>: >> >>>> >> >>>> Thanks. >> >>>> >> >>>> Decreasing the max deviation to 2 or 1 should help in your case. This >> >>>> option controls when the balancer stops trying to move PGs around -- by >> >>>> default it stops when the deviation from the mean is 5. Yes this is too >> >>>> large IMO -- all of our clusters have this set to 1. >> >>>> >> >>>> And given that you have some OSDs with more than 200 PGs, you >> >>>> definitely shouldn't increase the num PGs. >> >>>> >> >>>> But anyway with your mixed device sizes it might be challenging to make >> >>>> a perfectly uniform distribution. Give it a try with 1 though, and let >> >>>> us know how it goes. >> >>>> >> >>>> .. Dan >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> On Sat, Mar 13, 2021, 4:11 PM Boris Behrens <b...@kervyn.de> wrote: >> >>>>> >> >>>>> Hi Dan, >> >>>>> >> >>>>> upmap_max_deviation is default (5) in our cluster. Is 1 the >> >>>>> recommended deviation? >> >>>>> >> >>>>> I added the whole ceph osd df tree, (I need to remove some OSDs and >> >>>>> readd them as bluestore with SSD, so 69, 73 and 82 are a bit off now. >> >>>>> I also reweighted to try to get the %USE mitigated). >> >>>>> >> >>>>> I will increase the mgr debugging to see what is the problem. >> >>>>> >> >>>>> [root@s3db1 ~]# ceph osd df tree >> >>>>> ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META >> >>>>> AVAIL %USE VAR PGS STATUS TYPE NAME >> >>>>> -1 673.54224 - 659 TiB 491 TiB 464 TiB 96 GiB 1.2 TiB >> >>>>> 168 TiB 74.57 1.00 - root default >> >>>>> -2 58.30331 - 44 TiB 22 TiB 17 TiB 5.7 GiB 38 GiB >> >>>>> 22 TiB 49.82 0.67 - host s3db1 >> >>>>> 23 hdd 14.65039 1.00000 15 TiB 1.8 TiB 1.7 TiB 156 MiB 4.4 GiB >> >>>>> 13 TiB 12.50 0.17 101 up osd.23 >> >>>>> 69 hdd 14.55269 0 0 B 0 B 0 B 0 B 0 B >> >>>>> 0 B 0 0 11 up osd.69 >> >>>>> 73 hdd 14.55269 1.00000 15 TiB 10 TiB 10 TiB 6.1 MiB 33 GiB >> >>>>> 4.2 TiB 71.15 0.95 107 up osd.73 >> >>>>> 79 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 747 GiB 2.0 GiB 0 B >> >>>>> 747 GiB 79.94 1.07 52 up osd.79 >> >>>>> 80 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.0 TiB 1.9 GiB 0 B >> >>>>> 1.0 TiB 71.61 0.96 58 up osd.80 >> >>>>> 81 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.5 TiB 1.1 GiB 0 B >> >>>>> 1.5 TiB 60.07 0.81 55 up osd.81 >> >>>>> 82 hdd 3.63689 1.00000 3.6 TiB 1.9 TiB 1.7 TiB 536 MiB 0 B >> >>>>> 1.7 TiB 52.68 0.71 30 up osd.82 >> >>>>> -11 50.94173 - 51 TiB 38 TiB 38 TiB 3.7 GiB 100 GiB >> >>>>> 13 TiB 74.69 1.00 - host s3db10 >> >>>>> 63 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 616 MiB 14 GiB >> >>>>> 1.7 TiB 76.04 1.02 92 up osd.63 >> >>>>> 64 hdd 7.27739 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 820 MiB 15 GiB >> >>>>> 1.8 TiB 75.54 1.01 101 up osd.64 >> >>>>> 65 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 109 MiB 14 GiB >> >>>>> 2.0 TiB 73.17 0.98 105 up osd.65 >> >>>>> 66 hdd 7.27739 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 423 MiB 15 GiB >> >>>>> 1.4 TiB 80.38 1.08 98 up osd.66 >> >>>>> 67 hdd 7.27739 1.00000 7.3 TiB 5.1 TiB 5.1 TiB 572 MiB 14 GiB >> >>>>> 2.2 TiB 70.10 0.94 100 up osd.67 >> >>>>> 68 hdd 7.27739 1.00000 7.3 TiB 5.3 TiB 5.3 TiB 630 MiB 13 GiB >> >>>>> 2.0 TiB 72.88 0.98 107 up osd.68 >> >>>>> 70 hdd 7.27739 1.00000 7.3 TiB 5.4 TiB 5.4 TiB 648 MiB 14 GiB >> >>>>> 1.8 TiB 74.73 1.00 102 up osd.70 >> >>>>> -12 50.99052 - 51 TiB 39 TiB 39 TiB 2.9 GiB 99 GiB >> >>>>> 12 TiB 77.24 1.04 - host s3db11 >> >>>>> 46 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 102 MiB 15 GiB >> >>>>> 1.5 TiB 78.91 1.06 97 up osd.46 >> >>>>> 47 hdd 7.27739 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 61 MiB 13 GiB >> >>>>> 2.1 TiB 71.47 0.96 96 up osd.47 >> >>>>> 48 hdd 7.27739 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 853 MiB 15 GiB >> >>>>> 1.2 TiB 83.46 1.12 109 up osd.48 >> >>>>> 49 hdd 7.27739 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 708 MiB 15 GiB >> >>>>> 1.5 TiB 78.96 1.06 98 up osd.49 >> >>>>> 50 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 472 MiB 15 GiB >> >>>>> 1.4 TiB 80.40 1.08 102 up osd.50 >> >>>>> 51 hdd 7.27739 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 729 MiB 15 GiB >> >>>>> 1.3 TiB 81.70 1.10 110 up osd.51 >> >>>>> 72 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 91 MiB 12 GiB >> >>>>> 2.5 TiB 65.82 0.88 89 up osd.72 >> >>>>> -37 58.55478 - 59 TiB 46 TiB 46 TiB 5.0 GiB 124 GiB >> >>>>> 12 TiB 79.04 1.06 - host s3db12 >> >>>>> 19 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 462 MiB 8.2 GiB >> >>>>> 559 GiB 85.18 1.14 55 up osd.19 >> >>>>> 71 hdd 3.68750 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 3.9 MiB 7.8 GiB >> >>>>> 825 GiB 78.14 1.05 50 up osd.71 >> >>>>> 75 hdd 3.68750 1.00000 3.7 TiB 3.1 TiB 3.1 TiB 576 MiB 8.3 GiB >> >>>>> 555 GiB 85.29 1.14 57 up osd.75 >> >>>>> 76 hdd 3.68750 1.00000 3.7 TiB 3.2 TiB 3.1 TiB 239 MiB 9.3 GiB >> >>>>> 501 GiB 86.73 1.16 50 up osd.76 >> >>>>> 77 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 880 MiB 30 GiB >> >>>>> 3.6 TiB 75.57 1.01 202 up osd.77 >> >>>>> 78 hdd 14.60159 1.00000 15 TiB 11 TiB 11 TiB 1.0 GiB 30 GiB >> >>>>> 3.4 TiB 76.65 1.03 196 up osd.78 >> >>>>> 83 hdd 14.60159 1.00000 15 TiB 12 TiB 12 TiB 1.8 GiB 31 GiB >> >>>>> 2.9 TiB 80.04 1.07 223 up osd.83 >> >>>>> -3 58.49872 - 58 TiB 43 TiB 38 TiB 8.1 GiB 91 GiB >> >>>>> 16 TiB 73.15 0.98 - host s3db2 >> >>>>> 1 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 3.1 GiB 38 GiB >> >>>>> 3.6 TiB 75.52 1.01 194 up osd.1 >> >>>>> 3 hdd 3.63689 1.00000 3.6 TiB 2.2 TiB 1.4 TiB 418 MiB 0 B >> >>>>> 1.4 TiB 60.94 0.82 52 up osd.3 >> >>>>> 4 hdd 3.63689 0.89999 3.6 TiB 3.2 TiB 401 GiB 845 MiB 0 B >> >>>>> 401 GiB 89.23 1.20 53 up osd.4 >> >>>>> 5 hdd 3.63689 1.00000 3.6 TiB 2.3 TiB 1.3 TiB 437 MiB 0 B >> >>>>> 1.3 TiB 62.88 0.84 51 up osd.5 >> >>>>> 6 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 1.8 GiB 0 B >> >>>>> 1.7 TiB 54.51 0.73 47 up osd.6 >> >>>>> 7 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 493 MiB 26 GiB >> >>>>> 3.8 TiB 73.90 0.99 185 up osd.7 >> >>>>> 74 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.1 GiB 27 GiB >> >>>>> 3.5 TiB 76.27 1.02 208 up osd.74 >> >>>>> -4 58.49872 - 58 TiB 43 TiB 37 TiB 33 GiB 86 GiB >> >>>>> 15 TiB 74.05 0.99 - host s3db3 >> >>>>> 2 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 850 MiB 26 GiB >> >>>>> 4.0 TiB 72.78 0.98 203 up osd.2 >> >>>>> 9 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 8.3 GiB 33 GiB >> >>>>> 3.6 TiB 75.62 1.01 189 up osd.9 >> >>>>> 10 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 663 MiB 28 GiB >> >>>>> 3.5 TiB 76.34 1.02 211 up osd.10 >> >>>>> 12 hdd 3.63689 1.00000 3.6 TiB 2.4 TiB 1.2 TiB 633 MiB 0 B >> >>>>> 1.2 TiB 66.22 0.89 44 up osd.12 >> >>>>> 13 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 720 GiB 2.3 GiB 0 B >> >>>>> 720 GiB 80.66 1.08 66 up osd.13 >> >>>>> 14 hdd 3.63689 1.00000 3.6 TiB 3.1 TiB 552 GiB 18 GiB 0 B >> >>>>> 552 GiB 85.18 1.14 60 up osd.14 >> >>>>> 15 hdd 3.63689 1.00000 3.6 TiB 2.0 TiB 1.7 TiB 2.1 GiB 0 B >> >>>>> 1.7 TiB 53.72 0.72 44 up osd.15 >> >>>>> -5 58.49872 - 58 TiB 45 TiB 37 TiB 7.2 GiB 99 GiB >> >>>>> 14 TiB 76.37 1.02 - host s3db4 >> >>>>> 11 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 897 MiB 28 GiB >> >>>>> 2.8 TiB 81.15 1.09 205 up osd.11 >> >>>>> 17 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 27 GiB >> >>>>> 3.6 TiB 75.38 1.01 211 up osd.17 >> >>>>> 18 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 965 MiB 44 GiB >> >>>>> 4.0 TiB 72.86 0.98 188 up osd.18 >> >>>>> 20 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 796 GiB 529 MiB 0 B >> >>>>> 796 GiB 78.63 1.05 66 up osd.20 >> >>>>> 21 hdd 3.63689 1.00000 3.6 TiB 2.6 TiB 1.1 TiB 2.1 GiB 0 B >> >>>>> 1.1 TiB 70.32 0.94 47 up osd.21 >> >>>>> 22 hdd 3.63689 1.00000 3.6 TiB 2.9 TiB 802 GiB 882 MiB 0 B >> >>>>> 802 GiB 78.47 1.05 58 up osd.22 >> >>>>> 24 hdd 3.63689 1.00000 3.6 TiB 2.8 TiB 856 GiB 645 MiB 0 B >> >>>>> 856 GiB 77.01 1.03 47 up osd.24 >> >>>>> -6 58.89636 - 59 TiB 44 TiB 44 TiB 2.4 GiB 111 GiB >> >>>>> 15 TiB 75.22 1.01 - host s3db5 >> >>>>> 0 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 70 MiB 6.6 GiB >> >>>>> 1.3 TiB 65.00 0.87 48 up osd.0 >> >>>>> 25 hdd 3.73630 1.00000 3.7 TiB 2.4 TiB 2.3 TiB 5.3 MiB 6.6 GiB >> >>>>> 1.4 TiB 63.86 0.86 41 up osd.25 >> >>>>> 26 hdd 3.73630 1.00000 3.7 TiB 2.9 TiB 2.8 TiB 181 MiB 7.6 GiB >> >>>>> 862 GiB 77.47 1.04 48 up osd.26 >> >>>>> 27 hdd 3.73630 1.00000 3.7 TiB 2.3 TiB 2.2 TiB 7.0 MiB 6.1 GiB >> >>>>> 1.5 TiB 61.00 0.82 48 up osd.27 >> >>>>> 28 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 937 MiB 30 GiB >> >>>>> 2.8 TiB 81.19 1.09 203 up osd.28 >> >>>>> 29 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 536 MiB 26 GiB >> >>>>> 3.8 TiB 73.95 0.99 200 up osd.29 >> >>>>> 30 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 744 MiB 28 GiB >> >>>>> 3.1 TiB 79.07 1.06 207 up osd.30 >> >>>>> -7 58.89636 - 59 TiB 44 TiB 44 TiB 14 GiB 122 GiB >> >>>>> 14 TiB 75.41 1.01 - host s3db6 >> >>>>> 32 hdd 3.73630 1.00000 3.7 TiB 3.1 TiB 3.0 TiB 16 MiB 8.2 GiB >> >>>>> 622 GiB 83.74 1.12 65 up osd.32 >> >>>>> 33 hdd 3.73630 0.79999 3.7 TiB 3.0 TiB 2.9 TiB 14 MiB 8.1 GiB >> >>>>> 740 GiB 80.67 1.08 52 up osd.33 >> >>>>> 34 hdd 3.73630 0.79999 3.7 TiB 2.9 TiB 2.8 TiB 449 MiB 7.7 GiB >> >>>>> 877 GiB 77.08 1.03 52 up osd.34 >> >>>>> 35 hdd 3.73630 0.79999 3.7 TiB 2.3 TiB 2.2 TiB 133 MiB 7.0 GiB >> >>>>> 1.4 TiB 62.18 0.83 42 up osd.35 >> >>>>> 36 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 544 MiB 26 GiB >> >>>>> 4.0 TiB 72.98 0.98 220 up osd.36 >> >>>>> 37 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 11 GiB 38 GiB >> >>>>> 3.6 TiB 75.30 1.01 200 up osd.37 >> >>>>> 38 hdd 14.65039 1.00000 15 TiB 11 TiB 11 TiB 1.2 GiB 28 GiB >> >>>>> 3.3 TiB 77.43 1.04 217 up osd.38 >> >>>>> -8 58.89636 - 59 TiB 47 TiB 46 TiB 3.9 GiB 116 GiB >> >>>>> 12 TiB 78.98 1.06 - host s3db7 >> >>>>> 39 hdd 3.73630 1.00000 3.7 TiB 3.2 TiB 3.2 TiB 19 MiB 8.5 GiB >> >>>>> 499 GiB 86.96 1.17 43 up osd.39 >> >>>>> 40 hdd 3.73630 1.00000 3.7 TiB 2.6 TiB 2.5 TiB 144 MiB 7.0 GiB >> >>>>> 1.2 TiB 68.33 0.92 39 up osd.40 >> >>>>> 41 hdd 3.73630 1.00000 3.7 TiB 3.0 TiB 2.9 TiB 218 MiB 7.9 GiB >> >>>>> 732 GiB 80.86 1.08 64 up osd.41 >> >>>>> 42 hdd 3.73630 1.00000 3.7 TiB 2.5 TiB 2.4 TiB 594 MiB 7.0 GiB >> >>>>> 1.2 TiB 67.97 0.91 50 up osd.42 >> >>>>> 43 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 564 MiB 28 GiB >> >>>>> 2.9 TiB 80.32 1.08 213 up osd.43 >> >>>>> 44 hdd 14.65039 1.00000 15 TiB 12 TiB 11 TiB 1.3 GiB 28 GiB >> >>>>> 3.1 TiB 78.59 1.05 198 up osd.44 >> >>>>> 45 hdd 14.65039 1.00000 15 TiB 12 TiB 12 TiB 1.2 GiB 30 GiB >> >>>>> 2.8 TiB 81.05 1.09 214 up osd.45 >> >>>>> -9 51.28331 - 51 TiB 41 TiB 41 TiB 4.9 GiB 108 GiB >> >>>>> 10 TiB 79.75 1.07 - host s3db8 >> >>>>> 8 hdd 7.32619 1.00000 7.3 TiB 5.8 TiB 5.8 TiB 472 MiB 15 GiB >> >>>>> 1.5 TiB 79.68 1.07 99 up osd.8 >> >>>>> 16 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.8 TiB 785 MiB 15 GiB >> >>>>> 1.4 TiB 80.25 1.08 97 up osd.16 >> >>>>> 31 hdd 7.32619 1.00000 7.3 TiB 5.5 TiB 5.5 TiB 438 MiB 14 GiB >> >>>>> 1.8 TiB 75.36 1.01 87 up osd.31 >> >>>>> 52 hdd 7.32619 1.00000 7.3 TiB 5.7 TiB 5.7 TiB 844 MiB 15 GiB >> >>>>> 1.6 TiB 78.19 1.05 113 up osd.52 >> >>>>> 53 hdd 7.32619 1.00000 7.3 TiB 6.2 TiB 6.1 TiB 792 MiB 18 GiB >> >>>>> 1.1 TiB 84.46 1.13 109 up osd.53 >> >>>>> 54 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 959 MiB 15 GiB >> >>>>> 1.7 TiB 76.73 1.03 115 up osd.54 >> >>>>> 55 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.1 TiB 699 MiB 16 GiB >> >>>>> 1.2 TiB 83.56 1.12 122 up osd.55 >> >>>>> -10 51.28331 - 51 TiB 39 TiB 39 TiB 4.7 GiB 100 GiB >> >>>>> 12 TiB 76.05 1.02 - host s3db9 >> >>>>> 56 hdd 7.32619 1.00000 7.3 TiB 5.2 TiB 5.2 TiB 840 MiB 13 GiB >> >>>>> 2.1 TiB 71.06 0.95 105 up osd.56 >> >>>>> 57 hdd 7.32619 1.00000 7.3 TiB 6.1 TiB 6.0 TiB 1.0 GiB 16 GiB >> >>>>> 1.2 TiB 83.17 1.12 102 up osd.57 >> >>>>> 58 hdd 7.32619 1.00000 7.3 TiB 6.0 TiB 5.9 TiB 43 MiB 15 GiB >> >>>>> 1.4 TiB 81.56 1.09 105 up osd.58 >> >>>>> 59 hdd 7.32619 1.00000 7.3 TiB 5.9 TiB 5.9 TiB 429 MiB 15 GiB >> >>>>> 1.4 TiB 80.64 1.08 94 up osd.59 >> >>>>> 60 hdd 7.32619 1.00000 7.3 TiB 5.4 TiB 5.3 TiB 226 MiB 14 GiB >> >>>>> 2.0 TiB 73.25 0.98 101 up osd.60 >> >>>>> 61 hdd 7.32619 1.00000 7.3 TiB 4.8 TiB 4.8 TiB 1.1 GiB 12 GiB >> >>>>> 2.5 TiB 65.84 0.88 103 up osd.61 >> >>>>> 62 hdd 7.32619 1.00000 7.3 TiB 5.6 TiB 5.6 TiB 1.0 GiB 15 GiB >> >>>>> 1.7 TiB 76.83 1.03 126 up osd.62 >> >>>>> TOTAL 674 TiB 501 TiB 473 TiB 96 GiB 1.2 TiB >> >>>>> 173 TiB 74.57 >> >>>>> MIN/MAX VAR: 0.17/1.20 STDDEV: 10.25 >> >>>>> >> >>>>> >> >>>>> >> >>>>> Am Sa., 13. März 2021 um 15:57 Uhr schrieb Dan van der Ster >> >>>>> <d...@vanderster.com>: >> >>>>>> >> >>>>>> No, increasing num PGs won't help substantially. >> >>>>>> >> >>>>>> Can you share the entire output of ceph osd df tree ? >> >>>>>> >> >>>>>> Did you already set >> >>>>>> >> >>>>>> ceph config set mgr mgr/balancer/upmap_max_deviation 1 >> >>>>>> >> >>>>>> >> >>>>>> ?? >> >>>>>> And I recommend debug_mgr 4/5 so you can see some basic upmap >> >>>>>> balancer logging. >> >>>>>> >> >>>>>> .. Dan >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> On Sat, Mar 13, 2021, 3:49 PM Boris Behrens <b...@kervyn.de> wrote: >> >>>>>>> >> >>>>>>> Hello people, >> >>>>>>> >> >>>>>>> I am still struggeling with the balancer >> >>>>>>> (https://www.mail-archive.com/ceph-users@ceph.io/msg09124.html) >> >>>>>>> Now I've read some more and might think that I do not have enough >> >>>>>>> PGs. >> >>>>>>> Currently I have 84OSDs and 1024PGs for the main pool (3008 total). I >> >>>>>>> have the autoscaler enabled, but I doesn't tell me to increase the >> >>>>>>> PGs. >> >>>>>>> >> >>>>>>> What do you think? >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >> >>>>>>> im groüen Saal. >> >>>>>>> _______________________________________________ >> >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >> >>>>>>> To unsubscribe send an email to ceph-users-le...@ceph.io >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend >> >>>>> im groüen Saal. >> >>> >> >>> >> >>> >> >>> -- >> >>> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> >>> groüen Saal. >> > >> > >> > >> > -- >> > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> > groüen Saal. > > > > -- > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > groüen Saal. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
