First I tried with osd reweight, waited a few hours then osd crush
reweight, then with pg-umpap from Laimis. Seems to crush reweight was most
effective, but not for "all" osds I tried.
Uh, probably I've set ceph config set osd osd_max_backfills to high number
in the past, probably better to reduce it to 1 in steps, since now much
backfilling is already going on?
Output of commands in attachment.
Rok
On Sun, Dec 22, 2024 at 7:41 PM Alwin Antreich <alwin.antre...@croit.io>
wrote:
> Hi Rok,
>
> On Sun, 22 Dec 2024 at 16:08, Rok Jaklič <rjak...@gmail.com> wrote:
>
>> Thank you all for your suggestions.
>>
>> I've increased full ratio to 0.96 and rgw started to work again.
>>
>> However:
>>
>> I've tried to set for e.g. osd.122 reweight, crush reweight and then also
>> with:
>> ceph osd pg-upmap-items 9.169 122 187
>>
> A little too much at once. ;)
>
> 1) ceph osd reweight
> This overrules (lika a bias) the OSD weight temporarily but it will
> keep the crush weight of the bucket the same. Which means Ceph will try to
> keep the PGs within the node.
>
> 2) ceph osd crush reweight
> This sets the weight of the item in crush, which also tells Ceph that
> the node has a different weight and starts a new calculation for all the
> PGs connected to OSDs for this node. Which means more data movement.
>
> 3) ceph osd pg-upmap-items
> Overrides the placement of a PG specified by the crush algorithm. You
> can see these upmap entries in `ceph osd dump`.
>
> I do not recommend 1). With 3) you have the most control over where PGs
> should go. And it is best to keep the balancer off, otherwise it may
> interfere with your placements.
>
>
>>
>> and then also
>> [root@ctplmon1 plankton-swarm]# bash ./plankton-swarm.sh source-osds
>> osd.122 3
>> Using custom source OSDs: osd.122
>> Underused OSDs (<65%):
>> 63,108,200,199,143,144,198,142,140,146,195,141,148,194,196,197,103,158,191,147,125,25,164,54,193,126,192,145,46,19,68,157,128,15,60,53,134,129,87,0,102,131,6,43,78,127,120,33,81,119,3,5,74,190,70,8,160,24,58,156,114,29,186,82,96,116,182,48,84,28,18,44,178,39,4,75,115,76,79,130,72,86,159,40,184,22,26,35,71,171,88,64,175,187,170,165,41,110,94,150,111,17,83,14,27,49,2,37,124,172,177,98,152,104,95,118,168,189,132,52,105,32,59,21,101,9,107
>> Will now find ways to move 3 pgs in each OSD respecting node failure
>> domain.
>> Processing OSD osd.122...
>> dumped all
>> No active and clean pgs found for osd.122, skipping.
>> Balance pgs commands written to swarm-file - review and let planktons
>> swarm with 'bash swarm-file'.
>>
> On another post of yours you show 6x PGs being processed for osd.122,
> which is usually quite a lot for HDDs. Reducing the amount of
> osd_max_backfills (for wpq) to 1 will reduce the amount of parallel
> backfills. Where at most 2x backfills per OSD will happen.
> This might reduce the extra space needed during rebalance of these PGs, as
> the time needed to complete the rebalance of a particular PG will only
> increase the more PGs are processed in parallel. Once the PG has backfilled
> successfully it will be removed from the old OSD.
>
> You can also try to use `ceph pg force-backfill` of particular PGs to let
> Ceph know to process them prior to other PGs.
>
> Can you please post a `ceph osd df tree` and a `ceph df` to give us a
> picture on how things are?
>
> Cheers,
> Alwin
> croit GmbH, https://croit.io
>
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 1051.48230 - 1.0 PiB 611 TiB 605 TiB 385 GiB 4.9 TiB
433 TiB 58.53 1.00 - root default
-3 153.27872 - 154 TiB 90 TiB 89 TiB 98 GiB 751 GiB
64 TiB 58.34 1.00 - host ctplosd1
0 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.6 TiB 19 MiB 21 GiB
2.9 TiB 47.55 0.81 18 up osd.0
1 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 6 KiB 30 GiB
1.9 TiB 64.33 1.10 18 up osd.1
2 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 13 MiB 28 GiB
2.1 TiB 61.93 1.06 17 up osd.2
3 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.5 TiB 8.4 MiB 23 GiB
3.0 TiB 45.64 0.78 14 up osd.3
4 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 93 MiB 25 GiB
2.4 TiB 56.93 0.97 18 up osd.4
5 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.7 TiB 44 KiB 22 GiB
2.7 TiB 50.24 0.86 13 up osd.5
6 hdd 5.45798 1.00000 5.5 TiB 2.4 TiB 2.4 TiB 6 KiB 20 GiB
3.0 TiB 44.26 0.76 13 up osd.6
7 hdd 5.45798 1.00000 5.5 TiB 4.5 TiB 4.4 TiB 15 MiB 35 GiB
1007 GiB 81.97 1.40 28 up osd.7
8 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 3.9 MiB 23 GiB
2.6 TiB 52.16 0.89 19 up osd.8
9 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 146 KiB 27 GiB
2.1 TiB 60.71 1.04 16 up osd.9
10 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 8 KiB 30 GiB
1.9 TiB 64.55 1.10 17 up osd.10
11 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 41 KiB 30 GiB
1.8 TiB 66.51 1.14 23 up osd.11
12 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.6 TiB 19 MiB 30 GiB
1.8 TiB 66.91 1.14 21 up osd.12
13 hdd 4.45798 0.59999 5.5 TiB 4.9 TiB 4.9 TiB 8 KiB 38 GiB
527 GiB 90.57 1.55 23 up osd.13
14 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 3.7 MiB 28 GiB
2.1 TiB 60.96 1.04 15 up osd.14
15 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.4 TiB 65 KiB 20 GiB
3.0 TiB 45.13 0.77 13 up osd.15
16 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 31 KiB 29 GiB
1.7 TiB 68.14 1.16 20 up osd.16
17 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.2 TiB 24 MiB 27 GiB
2.2 TiB 59.91 1.02 20 up osd.17
18 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 3.1 MiB 25 GiB
2.6 TiB 53.22 0.91 19 up osd.18
19 hdd 5.45798 1.00000 5.5 TiB 1.7 TiB 1.7 TiB 49 KiB 17 GiB
3.8 TiB 31.06 0.53 13 up osd.19
20 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 6 KiB 30 GiB
1.8 TiB 66.39 1.13 22 up osd.20
21 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 4.1 MiB 29 GiB
1.8 TiB 66.20 1.13 17 up osd.21
22 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.1 TiB 7.5 MiB 25 GiB
2.3 TiB 58.02 0.99 20 up osd.22
23 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 5.7 MiB 34 GiB
1.0 TiB 81.35 1.39 24 up osd.23
24 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 14 MiB 23 GiB
2.6 TiB 52.92 0.90 17 up osd.24
25 hdd 5.45798 1.00000 5.5 TiB 1.8 TiB 1.8 TiB 45 KiB 15 GiB
3.6 TiB 33.27 0.57 10 up osd.25
26 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 5.5 MiB 26 GiB
2.5 TiB 54.80 0.94 19 up osd.26
27 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 2 KiB 27 GiB
2.1 TiB 61.52 1.05 17 up osd.27
140 ssd 0.72768 1.00000 745 GiB 40 GiB 543 MiB 34 GiB 5.4 GiB
706 GiB 5.32 0.09 4 up osd.140
141 ssd 0.72768 1.00000 745 GiB 72 GiB 519 MiB 64 GiB 8.1 GiB
673 GiB 9.72 0.17 5 up osd.141
-28 16.37398 - 16 TiB 2.3 TiB 2.3 TiB 61 KiB 21 GiB
14 TiB 14.14 0.24 - host ctplosd2
198 hdd 5.45799 1.00000 5.5 TiB 893 GiB 885 GiB 53 KiB 7.9 GiB
4.6 TiB 15.97 0.27 3 up osd.198
199 hdd 5.45799 1.00000 5.5 TiB 789 GiB 782 GiB 2 KiB 6.8 GiB
4.7 TiB 14.11 0.24 7 up osd.199
200 hdd 5.45799 1.00000 5.5 TiB 690 GiB 684 GiB 6 KiB 5.8 GiB
4.8 TiB 12.34 0.21 1 up osd.200
-25 114.61771 - 115 TiB 60 TiB 59 TiB 158 MiB 486 GiB
55 TiB 52.16 0.89 - host ctplosd3
177 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 12 MiB 28 GiB
1.9 TiB 64.84 1.11 19 up osd.177
178 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.7 TiB 5.9 MiB 22 GiB
2.7 TiB 49.90 0.85 15 up osd.178
179 hdd 5.45798 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 8 KiB 33 GiB
1.3 TiB 75.55 1.29 17 up osd.179
180 hdd 5.45798 0.70000 5.5 TiB 4.0 TiB 4.0 TiB 14 MiB 38 GiB
1.4 TiB 74.15 1.27 22 up osd.180
181 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 11 MiB 32 GiB
1.5 TiB 71.81 1.23 16 up osd.181
182 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 5.7 MiB 25 GiB
2.3 TiB 57.54 0.98 17 up osd.182
183 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 5.4 MiB 30 GiB
1.8 TiB 67.91 1.16 18 up osd.183
184 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 261 KiB 25 GiB
2.4 TiB 56.90 0.97 12 up osd.184
185 hdd 5.45798 1.00000 5.5 TiB 4.1 TiB 4.0 TiB 38 MiB 33 GiB
1.4 TiB 74.53 1.27 22 up osd.185
186 hdd 5.45799 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 39 KiB 25 GiB
2.4 TiB 56.65 0.97 17 up osd.186
187 hdd 5.45799 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 4 KiB 25 GiB
2.3 TiB 57.34 0.98 16 up osd.187
188 hdd 5.45799 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 8 KiB 32 GiB
1.2 TiB 78.63 1.34 20 up osd.188
189 hdd 5.45799 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 4 KiB 29 GiB
1.8 TiB 66.34 1.13 12 up osd.189
190 hdd 5.45799 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 23 MiB 22 GiB
2.7 TiB 51.26 0.88 11 up osd.190
191 hdd 5.45799 1.00000 5.5 TiB 1.8 TiB 1.8 TiB 5.3 MiB 15 GiB
3.7 TiB 33.05 0.56 2 up osd.191
192 hdd 5.45799 1.00000 5.5 TiB 2.3 TiB 2.3 TiB 11 MiB 18 GiB
3.2 TiB 41.89 0.72 6 up osd.192
193 hdd 5.45799 1.00000 5.5 TiB 2.2 TiB 2.2 TiB 4 KiB 18 GiB
3.2 TiB 40.86 0.70 5 up osd.193
194 hdd 5.45799 1.00000 5.5 TiB 1.1 TiB 1.1 TiB 10 MiB 9.8 GiB
4.4 TiB 20.07 0.34 4 up osd.194
195 hdd 5.45799 1.00000 5.5 TiB 726 GiB 720 GiB 10 MiB 6.2 GiB
4.7 TiB 12.98 0.22 4 up osd.195
196 hdd 5.45799 1.00000 5.5 TiB 1.1 TiB 1.1 TiB 6 KiB 9.3 GiB
4.4 TiB 19.46 0.33 4 up osd.196
197 hdd 5.45799 1.00000 5.5 TiB 1.3 TiB 1.3 TiB 5.6 MiB 10 GiB
4.2 TiB 23.77 0.41 4 up osd.197
-22 152.82379 - 153 TiB 94 TiB 93 TiB 135 MiB 764 GiB
59 TiB 61.38 1.05 - host ctplosd4
149 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.4 MiB 27 GiB
2.0 TiB 62.97 1.08 21 up osd.149
150 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 19 MiB 27 GiB
2.2 TiB 59.42 1.02 19 up osd.150
151 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 6 KiB 33 GiB
1.2 TiB 77.98 1.33 24 up osd.151
152 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 5.7 MiB 26 GiB
2.2 TiB 59.36 1.01 15 up osd.152
153 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 14 MiB 29 GiB
1.8 TiB 67.64 1.16 22 up osd.153
154 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 2 KiB 35 GiB
1.1 TiB 79.23 1.35 20 up osd.154
155 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 23 MiB 36 GiB
1.1 TiB 80.66 1.38 25 up osd.155
156 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 6 KiB 23 GiB
2.7 TiB 50.99 0.87 16 up osd.156
157 hdd 5.45798 1.00000 5.5 TiB 2.2 TiB 2.2 TiB 6 KiB 20 GiB
3.2 TiB 40.46 0.69 11 up osd.157
158 hdd 5.45798 1.00000 5.5 TiB 1.5 TiB 1.5 TiB 2 KiB 14 GiB
3.9 TiB 28.11 0.48 11 up osd.158
159 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.1 TiB 4 KiB 26 GiB
2.3 TiB 57.97 0.99 22 up osd.159
160 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.6 TiB 38 KiB 23 GiB
2.8 TiB 48.87 0.84 21 up osd.160
161 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 33 KiB 29 GiB
1.9 TiB 65.68 1.12 15 up osd.161
162 hdd 5.45798 1.00000 5.5 TiB 3.8 TiB 3.8 TiB 3.9 MiB 33 GiB
1.7 TiB 69.74 1.19 19 up osd.162
163 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.3 TiB 4.4 MiB 34 GiB
1.1 TiB 79.11 1.35 21 up osd.163
164 hdd 5.45798 1.00000 5.5 TiB 1.7 TiB 1.7 TiB 6 KiB 15 GiB
3.7 TiB 31.95 0.55 10 up osd.164
165 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.0 TiB 4.2 MiB 25 GiB
2.4 TiB 55.93 0.96 18 up osd.165
166 hdd 5.45798 1.00000 5.5 TiB 3.8 TiB 3.8 TiB 8 KiB 30 GiB
1.7 TiB 69.70 1.19 21 up osd.166
167 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.8 TiB 8.9 MiB 32 GiB
1.6 TiB 71.09 1.21 18 up osd.167
168 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 2 KiB 27 GiB
2.0 TiB 64.12 1.10 18 up osd.168
169 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 7.6 MiB 33 GiB
1.7 TiB 68.01 1.16 23 up osd.169
170 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 8.8 MiB 25 GiB
2.4 TiB 55.31 0.94 20 up osd.170
171 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 34 KiB 25 GiB
2.2 TiB 58.84 1.01 18 up osd.171
172 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6.5 MiB 27 GiB
2.0 TiB 63.13 1.08 17 up osd.172
173 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 8.1 MiB 27 GiB
2.1 TiB 61.48 1.05 16 up osd.173
174 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 6 KiB 30 GiB
1.6 TiB 71.32 1.22 21 up osd.174
175 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.0 TiB 54 KiB 26 GiB
2.4 TiB 56.02 0.96 20 up osd.175
176 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.4 TiB 14 MiB 28 GiB
2.0 TiB 63.55 1.09 20 up osd.176
-19 0 - 0 B 0 B 0 B 0 B 0 B
0 B 0 0 - host ctplosd4.arnes.si
-11 153.27872 - 154 TiB 93 TiB 92 TiB 64 GiB 772 GiB
61 TiB 60.15 1.03 - host ctplosd5
112 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.3 TiB 32 KiB 35 GiB
1.1 TiB 80.13 1.37 18 up osd.112
113 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 25 KiB 30 GiB
1.8 TiB 67.74 1.16 23 up osd.113
114 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 46 KiB 24 GiB
2.7 TiB 50.86 0.87 15 up osd.114
115 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 13 MiB 26 GiB
2.3 TiB 57.04 0.97 20 up osd.115
116 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 5.5 MiB 24 GiB
2.5 TiB 54.75 0.94 15 up osd.116
117 hdd 5.45798 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 10 KiB 31 GiB
1.5 TiB 73.12 1.25 21 up osd.117
118 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 12 MiB 28 GiB
1.9 TiB 64.36 1.10 19 up osd.118
119 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.7 TiB 12 MiB 23 GiB
2.7 TiB 49.83 0.85 16 up osd.119
120 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.7 TiB 51 KiB 22 GiB
2.7 TiB 50.47 0.86 12 up osd.120
121 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 6 KiB 32 GiB
1.5 TiB 72.00 1.23 23 up osd.121
122 hdd 4.45798 0.59999 5.5 TiB 4.7 TiB 4.7 TiB 16 KiB 37 GiB
743 GiB 86.70 1.48 18 up osd.122
123 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 10 MiB 33 GiB
1.5 TiB 72.07 1.23 22 up osd.123
124 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 5.8 MiB 27 GiB
2.1 TiB 62.34 1.07 17 up osd.124
125 hdd 5.45798 1.00000 5.5 TiB 1.5 TiB 1.5 TiB 5.6 MiB 14 GiB
3.9 TiB 28.07 0.48 11 up osd.125
126 hdd 5.45798 1.00000 5.5 TiB 1.9 TiB 1.9 TiB 9.2 MiB 18 GiB
3.5 TiB 35.26 0.60 13 up osd.126
127 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.6 TiB 7 KiB 22 GiB
2.8 TiB 48.17 0.82 14 up osd.127
128 hdd 5.45798 1.00000 5.5 TiB 2.4 TiB 2.4 TiB 2 KiB 22 GiB
3.0 TiB 44.65 0.76 17 up osd.128
129 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.5 TiB 11 MiB 23 GiB
2.9 TiB 46.46 0.79 22 up osd.129
130 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 18 MiB 24 GiB
2.5 TiB 53.84 0.92 21 up osd.130
131 hdd 5.45798 1.00000 5.5 TiB 2.4 TiB 2.4 TiB 7.5 MiB 21 GiB
3.0 TiB 44.17 0.75 15 up osd.131
132 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 6 KiB 25 GiB
2.7 TiB 50.91 0.87 10 up osd.132
133 hdd 5.45798 1.00000 5.5 TiB 4.5 TiB 4.4 TiB 28 MiB 35 GiB
1.0 TiB 81.59 1.39 25 up osd.133
134 hdd 5.45798 1.00000 5.5 TiB 2.3 TiB 2.3 TiB 10 KiB 20 GiB
3.1 TiB 42.36 0.72 17 up osd.134
135 hdd 5.45798 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 6 KiB 33 GiB
1.4 TiB 73.80 1.26 23 up osd.135
136 hdd 5.45798 1.00000 5.5 TiB 4.0 TiB 4.0 TiB 3.8 MiB 32 GiB
1.4 TiB 73.87 1.26 22 up osd.136
137 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 5.0 MiB 35 GiB
1.0 TiB 81.52 1.39 25 up osd.137
138 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 14 MiB 35 GiB
1.1 TiB 80.62 1.38 20 up osd.138
139 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.9 TiB 14 MiB 32 GiB
1.5 TiB 72.29 1.24 22 up osd.139
142 ssd 0.72768 1.00000 745 GiB 39 GiB 511 MiB 33 GiB 5.7 GiB
706 GiB 5.27 0.09 4 up osd.142
143 ssd 0.72768 1.00000 745 GiB 37 GiB 491 MiB 31 GiB 6.1 GiB
708 GiB 5.00 0.09 2 up osd.143
-7 154.27917 - 149 TiB 87 TiB 87 TiB 97 GiB 729 GiB
61 TiB 58.69 1.00 - host ctplosd6
57 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 4 KiB 29 GiB
1.8 TiB 67.86 1.16 16 up osd.57
58 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.8 TiB 4 KiB 23 GiB
2.6 TiB 52.49 0.90 16 up osd.58
59 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 2 KiB 29 GiB
1.9 TiB 64.95 1.11 19 up osd.59
60 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.5 TiB 18 KiB 21 GiB
3.0 TiB 45.69 0.78 16 up osd.60
61 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 4 KiB 28 GiB
1.8 TiB 66.27 1.13 22 up osd.61
62 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 32 KiB 28 GiB
2.2 TiB 59.06 1.01 25 up osd.62
63 hdd 5.45798 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 0 down osd.63
64 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 4 KiB 26 GiB
2.2 TiB 59.12 1.01 20 up osd.64
65 hdd 5.45798 1.00000 5.5 TiB 4.4 TiB 4.4 TiB 4.3 MiB 35 GiB
1.0 TiB 81.34 1.39 27 up osd.65
66 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 19 MiB 30 GiB
1.9 TiB 65.92 1.13 19 up osd.66
67 hdd 5.45798 1.00000 5.5 TiB 4.0 TiB 3.9 TiB 25 MiB 31 GiB
1.5 TiB 72.67 1.24 21 up osd.67
68 hdd 5.45798 1.00000 5.5 TiB 2.1 TiB 2.1 TiB 4 KiB 19 GiB
3.3 TiB 38.91 0.66 12 up osd.68
69 hdd 5.45798 1.00000 5.5 TiB 4.1 TiB 4.1 TiB 12 MiB 33 GiB
1.3 TiB 75.83 1.30 21 up osd.69
70 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 17 MiB 23 GiB
2.7 TiB 51.32 0.88 27 up osd.70
71 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 13 MiB 25 GiB
2.5 TiB 54.58 0.93 16 up osd.71
72 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.7 TiB 10 MiB 24 GiB
2.7 TiB 50.74 0.87 18 up osd.72
73 hdd 5.45798 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 4 KiB 33 GiB
1.3 TiB 76.92 1.31 20 up osd.73
74 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 6 KiB 23 GiB
2.7 TiB 50.93 0.87 15 up osd.74
75 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 47 KiB 26 GiB
2.3 TiB 57.52 0.98 18 up osd.75
76 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 16 MiB 24 GiB
2.5 TiB 53.94 0.92 17 up osd.76
77 hdd 5.45798 1.00000 5.5 TiB 3.8 TiB 3.8 TiB 51 MiB 31 GiB
1.6 TiB 70.08 1.20 23 up osd.77
78 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.6 TiB 10 KiB 22 GiB
2.8 TiB 48.25 0.82 14 up osd.78
79 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.7 TiB 6 KiB 23 GiB
2.7 TiB 49.88 0.85 19 up osd.79
80 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 9.2 MiB 32 GiB
1.9 TiB 65.97 1.13 21 up osd.80
81 hdd 5.45798 1.00000 5.5 TiB 2.7 TiB 2.7 TiB 8.9 MiB 23 GiB
2.8 TiB 49.41 0.84 17 up osd.81
82 hdd 5.45798 1.00000 5.5 TiB 2.8 TiB 2.8 TiB 14 MiB 24 GiB
2.6 TiB 51.51 0.88 16 up osd.82
83 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 7 KiB 26 GiB
2.3 TiB 58.56 1.00 19 up osd.83
84 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 19 KiB 24 GiB
2.5 TiB 55.02 0.94 18 up osd.84
144 ssd 0.72768 1.00000 745 GiB 37 GiB 491 MiB 32 GiB 5.3 GiB
708 GiB 5.03 0.09 2 up osd.144
145 ssd 0.72768 1.00000 745 GiB 272 GiB 196 GiB 65 GiB 10 GiB
474 GiB 36.44 0.62 6 up osd.145
-5 153.55148 - 154 TiB 91 TiB 91 TiB 65 GiB 753 GiB
62 TiB 59.48 1.02 - host ctplosd7
28 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 2 KiB 24 GiB
2.4 TiB 55.81 0.95 17 up osd.28
29 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 66 KiB 23 GiB
2.5 TiB 53.29 0.91 16 up osd.29
30 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 8.9 MiB 25 GiB
2.3 TiB 57.40 0.98 19 up osd.30
31 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 14 MiB 33 GiB
1.2 TiB 78.01 1.33 19 up osd.31
32 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 11 MiB 29 GiB
2.1 TiB 61.93 1.06 17 up osd.32
33 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.5 TiB 10 KiB 22 GiB
2.9 TiB 46.34 0.79 15 up osd.33
34 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 4 KiB 34 GiB
1.2 TiB 78.44 1.34 23 up osd.34
35 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.1 TiB 12 MiB 26 GiB
2.3 TiB 58.03 0.99 23 up osd.35
36 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.5 TiB 4.0 MiB 30 GiB
1.9 TiB 65.53 1.12 21 up osd.36
37 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.3 TiB 11 MiB 27 GiB
2.1 TiB 61.76 1.06 24 up osd.37
38 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.8 TiB 14 MiB 30 GiB
1.6 TiB 70.76 1.21 22 up osd.38
39 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 5 KiB 25 GiB
2.4 TiB 56.76 0.97 18 up osd.39
40 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 20 KiB 25 GiB
2.3 TiB 58.71 1.00 23 up osd.40
41 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.2 TiB 6 KiB 26 GiB
2.2 TiB 59.95 1.02 19 up osd.41
42 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 5.0 MiB 35 GiB
1.2 TiB 78.22 1.34 22 up osd.42
43 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.6 TiB 4 KiB 22 GiB
2.8 TiB 48.04 0.82 14 up osd.43
44 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 48 MiB 25 GiB
2.4 TiB 56.42 0.96 23 up osd.44
45 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 10 MiB 28 GiB
2.0 TiB 62.58 1.07 18 up osd.45
46 hdd 5.45798 1.00000 5.5 TiB 2.1 TiB 2.1 TiB 8.5 MiB 18 GiB
3.3 TiB 38.95 0.67 11 up osd.46
47 hdd 5.45798 1.00000 5.5 TiB 3.9 TiB 3.8 TiB 9 KiB 32 GiB
1.6 TiB 71.08 1.21 21 up osd.47
48 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 9 KiB 24 GiB
2.5 TiB 54.68 0.93 16 up osd.48
49 hdd 5.45798 1.00000 5.5 TiB 2.9 TiB 2.9 TiB 10 KiB 25 GiB
2.5 TiB 53.51 0.91 18 up osd.49
50 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 3.0 TiB 5.4 MiB 28 GiB
2.4 TiB 55.30 0.94 16 up osd.50
51 hdd 5.45798 1.00000 5.5 TiB 3.7 TiB 3.7 TiB 7.9 MiB 29 GiB
1.7 TiB 67.95 1.16 17 up osd.51
52 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 74 KiB 27 GiB
2.2 TiB 60.51 1.03 19 up osd.52
53 hdd 5.45798 1.00000 5.5 TiB 2.5 TiB 2.5 TiB 9.2 MiB 21 GiB
2.9 TiB 46.65 0.80 14 up osd.53
54 hdd 5.45798 1.00000 5.5 TiB 2.0 TiB 2.0 TiB 6 KiB 17 GiB
3.5 TiB 36.19 0.62 12 up osd.54
55 hdd 5.45798 0.70000 5.5 TiB 4.3 TiB 4.3 TiB 6.8 MiB 33 GiB
1.1 TiB 79.13 1.35 23 up osd.55
148 ssd 0.72768 1.00000 745 GiB 75 GiB 491 MiB 65 GiB 9.4 GiB
670 GiB 10.08 0.17 4 up osd.148
-9 153.27872 - 149 TiB 93 TiB 93 TiB 61 GiB 771 GiB
55 TiB 62.74 1.07 - host ctplosd8
56 hdd 5.45798 1.00000 5.5 TiB 4.2 TiB 4.2 TiB 19 MiB 34 GiB
1.2 TiB 77.81 1.33 24 up osd.56
85 hdd 5.45798 1.00000 5.5 TiB 4.1 TiB 4.0 TiB 8.2 MiB 34 GiB
1.4 TiB 74.22 1.27 26 up osd.85
86 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.1 TiB 12 MiB 26 GiB
2.3 TiB 57.91 0.99 21 up osd.86
87 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.5 TiB 9 KiB 22 GiB
2.9 TiB 46.94 0.80 16 up osd.87
88 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.0 TiB 38 KiB 25 GiB
2.4 TiB 56.20 0.96 19 up osd.88
89 hdd 4.45798 0.70000 5.5 TiB 4.7 TiB 4.6 TiB 2 KiB 37 GiB
814 GiB 85.44 1.46 23 up osd.89
90 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 4.2 MiB 29 GiB
1.8 TiB 66.24 1.13 21 up osd.90
91 hdd 5.45798 1.00000 5.5 TiB 4.1 TiB 4.0 TiB 11 KiB 32 GiB
1.4 TiB 74.57 1.27 19 up osd.91
92 hdd 5.45798 1.00000 5.5 TiB 4.3 TiB 4.2 TiB 13 MiB 33 GiB
1.2 TiB 78.01 1.33 24 up osd.92
93 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 3.7 MiB 28 GiB
1.9 TiB 66.05 1.13 18 up osd.93
94 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.0 TiB 41 MiB 26 GiB
2.4 TiB 56.12 0.96 22 up osd.94
95 hdd 5.45798 1.00000 5.5 TiB 3.2 TiB 3.2 TiB 7 KiB 26 GiB
2.2 TiB 59.46 1.02 18 up osd.95
96 hdd 5.45798 1.00000 5.5 TiB 3.0 TiB 2.9 TiB 27 MiB 25 GiB
2.5 TiB 54.17 0.93 17 up osd.96
97 hdd 5.45798 1.00000 5.5 TiB 3.8 TiB 3.8 TiB 4 KiB 30 GiB
1.6 TiB 69.86 1.19 20 up osd.97
98 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 19 MiB 27 GiB
2.4 TiB 56.93 0.97 20 up osd.98
99 hdd 5.45798 0.70000 5.5 TiB 4.1 TiB 4.1 TiB 32 KiB 33 GiB
1.3 TiB 75.97 1.30 22 up osd.99
100 hdd 5.45798 0.79999 5.5 TiB 3.5 TiB 3.5 TiB 8.5 MiB 32 GiB
1.9 TiB 64.89 1.11 19 up osd.100
101 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 6 KiB 27 GiB
2.1 TiB 62.09 1.06 20 up osd.101
102 hdd 5.45798 1.00000 5.5 TiB 2.6 TiB 2.6 TiB 8 KiB 21 GiB
2.8 TiB 47.81 0.82 18 up osd.102
103 hdd 5.45798 1.00000 5.5 TiB 1.1 TiB 1.1 TiB 6.5 MiB 12 GiB
4.3 TiB 20.87 0.36 8 up osd.103
104 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.3 TiB 4 KiB 27 GiB
2.2 TiB 60.32 1.03 17 up osd.104
105 hdd 5.45798 1.00000 5.5 TiB 3.4 TiB 3.4 TiB 18 KiB 27 GiB
2.1 TiB 62.36 1.07 17 up osd.105
106 hdd 5.45798 1.00000 5.5 TiB 3.5 TiB 3.5 TiB 7.7 MiB 28 GiB
1.9 TiB 65.02 1.11 16 up osd.106
107 hdd 5.45798 1.00000 5.5 TiB 3.6 TiB 3.6 TiB 58 KiB 31 GiB
1.8 TiB 66.78 1.14 15 up osd.107
108 hdd 5.45798 0 0 B 0 B 0 B 0 B 0 B
0 B 0 0 0 down osd.108
109 hdd 5.45798 1.00000 5.5 TiB 4.5 TiB 4.5 TiB 1.6 MiB 36 GiB
948 GiB 83.03 1.42 22 up osd.109
110 hdd 5.45798 1.00000 5.5 TiB 3.1 TiB 3.1 TiB 25 KiB 25 GiB
2.3 TiB 57.04 0.97 17 up osd.110
111 hdd 5.45798 1.00000 5.5 TiB 3.3 TiB 3.2 TiB 18 MiB 27 GiB
2.2 TiB 59.88 1.02 18 up osd.111
146 ssd 0.72768 1.00000 745 GiB 51 GiB 547 MiB 45 GiB 5.9 GiB
694 GiB 6.89 0.12 5 up osd.146
147 ssd 0.72768 1.00000 745 GiB 217 GiB 196 GiB 16 GiB 5.1 GiB
529 GiB 29.07 0.50 3 up osd.147
TOTAL 1.0 PiB 611 TiB 605 TiB 385 GiB 4.9 TiB
433 TiB 58.53
MIN/MAX VAR: 0.09/1.55 STDDEV: 17.72
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 1.0 PiB 427 TiB 610 TiB 610 TiB 58.82
ssd 6.5 TiB 5.7 TiB 841 GiB 841 GiB 12.53
TOTAL 1.0 PiB 433 TiB 611 TiB 611 TiB 58.53
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX
AVAIL
.mgr 1 1 271 MiB 151 813 MiB 0 23
TiB
.rgw.root 2 32 1.3 KiB 4 56 KiB 0 23
TiB
default.rgw.log 3 32 131 MiB 342 395 MiB 0 23
TiB
default.rgw.control 4 32 0 B 8 0 B 0 23
TiB
default.rgw.meta 5 8 1003 KiB 3.08k 35 MiB 0 23
TiB
default.rgw.buckets.index 6 8 128 GiB 11.60k 384 GiB 8.77 1.3
TiB
default.rgw.buckets.non-ec 7 32 322 MiB 2.47k 995 MiB 0 23
TiB
default.rgw.buckets.data 9 512 363 TiB 256.57M 605 TiB 89.70 42
TiB
cephfs_data 10 128 4.3 GiB 1.08k 13 GiB 0.02 23
TiB
cephfs_metadata 11 8 288 KiB 23 973 KiB 0 23
TiB
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
