Hello!
Thank You for Your answer,
Right now we have problems with cluster on NVMe so DB naturally is also on
NVMe. Last time we have issue on HDD+NVMe(for DB/WAL). Now we have mClock
scheduler. Generally we are able to snaptrim but it cost CPU resources. We can
snaptrim thousands of snapshots per day. But the main problem and question
still remain. Why we got without any reason 2.5M of snaps to remove. What
should we do. How to prevent. It is a chance that if 2.5M of snaps came in
seconds then it is possible to remove it in few seconds. Some db corruption
maybe easy to clean.
Current cluster:
version - 17.2.7
mclock scheduler
I attached also outputs from:
ceph -s
ceph osd df tree
ceph osd pool ls detail
ceph df
I have to add that in json there is no entry in removed_snaps but in normal
output in removed_snaps_queue we can see:
> pool 1 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
> rjenkins pg_num 8192 pgp_num 8192 autoscale_mode off last_change 667587 lfor
> 0/0/1211 > flags hashpspool,selfmanaged_snaps stripe_width 0
> target_size_ratio 0.5 application rbd
> removed_snaps_queue [58ae2~3,58af4~3,58cd5~5,58cdd~3
> ... million+ ...
> c59ae~3,4c59b6~3,4c59bc~3]
Similar malformed output is in attached `ceph df`. Pool of course have still
available space but commands shows `0%`.
Best Regards,
Lukasz Lucki Gomulka
________________________________
From: Eugen Block <[email protected]>
Sent: 03 November 2025 11:53:06
To: [email protected]
Subject: [ceph-users] Re: Snaptrim flood
Hi,
are you using mclock scheduler (default in Quincy)? Until Reef 18.2.4
there was a default value set for osd_snap_trim_cost (1M bytes) which
blocked snaptrims [0]. This was fixed in [1] and backported to Reef.
But it's unlikely that this was your issue in Octopus as mclock became
the default in Pacific, IIRC. Since Quincy is also EOL, I'd recommed
to update further, if possible.
Were you able to avoid OSD flapping with the nodown flag (ceph osd set
nodown)? This can help to keep the cluster more stable in such
situations.
Can you add some more details about your setup like:
ceph -s
ceph osd df tree
ceph osd pool ls detail
ceph df
Are you using HDD OSDs or HDDs with dedicated DB/WAL? How many
snapshots are you generating?
Regards,
Eugen
[0] https://tracker.ceph.com/issues/67702
[1] https://tracker.ceph.com/issues/63604
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
nvme 964 TiB 376 TiB 588 TiB 588 TiB 60.99
TOTAL 964 TiB 376 TiB 588 TiB 588 TiB 60.99
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
volumes 1 8192 201 TiB 55.94M 586 TiB 100.00 0 B
.mgr 2 1 525 KiB 285 1.5 MiB 100.00 0 B
cluster:
id: <fsid>
health: HEALTH_WARN
noscrub,nodeep-scrub flag(s) set
8193 pgs not deep-scrubbed in time
8193 pgs not scrubbed in time
snap trim queue for 8192 pg(s) >= 1024
(mon_osd_snap_trim_queue_warn_on)
services:
mon: 3 daemons, quorum mon-06-<fsid>,mon-07-<fsid>,mon-08-<fsid> (age 2d)
mgr: mgr-07-<fsid>(active, since 20h), standbys: mgr-08-<fsid>,
mgr-06-<fsid>
osd: 284 osds: 284 up (since 39h), 284 in (since 7d)
flags noscrub,nodeep-scrub
data:
pools: 2 pools, 8193 pgs
objects: 55.93M objects, 198 TiB
usage: 588 TiB used, 376 TiB / 964 TiB avail
pgs: 8047 active+clean+snaptrim_wait
144 active+clean+snaptrim
2 active+clean
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META
AVAIL %USE VAR PGS STATUS TYPE NAME
-1 960.21631 - 964 TiB 588 TiB 586 TiB 8.5 GiB 1.7 TiB
376 TiB 60.99 1.00 - root default
-8 318.94107 - 319 TiB 196 TiB 195 TiB 2.8 GiB 593 GiB
123 TiB 61.37 1.01 - rack <rack>
-19 78.03877 - 78 TiB 48 TiB 48 TiB 716 MiB 146 GiB
30 TiB 62.03 1.02 - host <host>
95 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 61.09 1.00 87 up osd.95
96 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.4 GiB
1.2 TiB 63.38 1.04 90 up osd.96
98 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 60.01 0.98 85 up osd.98
99 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 33 MiB 6.7 GiB
1.3 TiB 63.16 1.04 90 up osd.99
101 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 61.19 1.00 87 up osd.101
102 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.7 GiB
1.2 TiB 63.32 1.04 90 up osd.102
103 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.7 GiB
1.2 TiB 63.28 1.04 90 up osd.103
122 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.41 1.04 90 up osd.122
123 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.45 1.04 91 up osd.123
124 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.35 1.04 90 up osd.124
125 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.1 GiB
1.4 TiB 59.87 0.98 85 up osd.125
126 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.90 0.98 85 up osd.126
127 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.7 GiB
1.2 TiB 63.32 1.04 90 up osd.127
128 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.5 GiB
1.2 TiB 63.50 1.04 90 up osd.128
129 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 59.73 0.98 85 up osd.129
130 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.6 GiB
1.2 TiB 63.28 1.04 90 up osd.130
131 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.4 GiB
1.2 TiB 63.42 1.04 90 up osd.131
132 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.99 0.98 85 up osd.132
133 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 60.00 0.98 85 up osd.133
134 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.13 1.02 88 up osd.134
135 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.2 TiB 63.34 1.04 90 up osd.135
137 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.7 GiB
1.3 TiB 62.89 1.03 89 up osd.137
138 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.69 0.98 85 up osd.138
-7 81.43176 - 81 TiB 50 TiB 50 TiB 740 MiB 151 GiB
31 TiB 61.75 1.01 - host <host>
4 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.43 1.02 89 up osd.4
17 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 61.32 1.01 87 up osd.17
23 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.80 0.98 85 up osd.23
24 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.74 0.98 85 up osd.24
25 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.3 GiB
1.2 TiB 63.53 1.04 90 up osd.25
26 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.7 GiB
1.2 TiB 63.48 1.04 90 up osd.26
27 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.68 1.03 89 up osd.27
28 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.6 GiB
1.3 TiB 62.72 1.03 89 up osd.28
29 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.6 GiB
1.2 TiB 63.32 1.04 90 up osd.29
30 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.2 TiB 63.34 1.04 90 up osd.30
31 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.3 GiB
1.4 TiB 59.93 0.98 85 up osd.31
32 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 59.53 0.98 85 up osd.32
33 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 59.81 0.98 85 up osd.33
34 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 33 MiB 6.5 GiB
1.2 TiB 63.56 1.04 90 up osd.34
35 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.52 1.04 90 up osd.35
36 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.4 GiB
1.2 TiB 63.44 1.04 90 up osd.36
37 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 59.98 0.98 85 up osd.37
38 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.2 TiB 63.17 1.04 90 up osd.38
39 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.4 GiB
1.2 TiB 63.37 1.04 90 up osd.39
40 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 59.95 0.98 85 up osd.40
43 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.3 GiB
1.4 TiB 59.74 0.98 85 up osd.43
45 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 31 MiB 6.6 GiB
1.2 TiB 63.57 1.04 90 up osd.45
68 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.2 GiB
1.4 TiB 59.96 0.98 85 up osd.68
203 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 5.6 GiB
1.4 TiB 60.06 0.98 85 up osd.203
-29 78.03877 - 78 TiB 47 TiB 47 TiB 703 MiB 144 GiB
31 TiB 60.37 0.99 - host <host>
69 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 5.5 GiB
1.4 TiB 60.09 0.99 85 up osd.69
217 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.6 GiB
1.3 TiB 60.67 0.99 86 up osd.217
218 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.75 0.98 85 up osd.218
219 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.62 1.03 89 up osd.219
223 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.3 TiB 60.39 0.99 86 up osd.223
229 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 59.20 0.97 84 up osd.229
232 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.2 GiB
1.4 TiB 59.21 0.97 84 up osd.232
234 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.20 0.97 84 up osd.234
235 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.19 0.97 84 up osd.235
242 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.23 0.97 84 up osd.242
243 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.68 1.03 89 up osd.243
244 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.19 0.97 84 up osd.244
245 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.80 0.98 85 up osd.245
255 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.01 0.97 84 up osd.255
256 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 61.91 1.02 88 up osd.256
258 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.3 GiB
1.3 TiB 61.88 1.01 88 up osd.258
259 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.02 0.97 84 up osd.259
269 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 59.19 0.97 84 up osd.269
270 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 33 MiB 6.7 GiB
1.3 TiB 62.80 1.03 89 up osd.270
271 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 62.73 1.03 89 up osd.271
272 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.60 1.03 89 up osd.272
280 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 59.07 0.97 84 up osd.280
281 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 59.15 0.97 84 up osd.281
-25 81.43176 - 81 TiB 50 TiB 50 TiB 740 MiB 152 GiB
31 TiB 61.33 1.01 - host <host>
143 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.3 GiB
1.3 TiB 62.59 1.03 89 up osd.143
144 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.89 0.98 85 up osd.144
149 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 59.80 0.98 85 up osd.149
152 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.77 1.03 89 up osd.152
159 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.96 1.03 89 up osd.159
160 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 59.82 0.98 85 up osd.160
161 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 7.2 GiB
1.3 TiB 62.74 1.03 89 up osd.161
162 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.76 0.98 85 up osd.162
168 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.4 GiB
1.4 TiB 59.94 0.98 85 up osd.168
169 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.61 1.03 89 up osd.169
172 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.67 1.03 89 up osd.172
175 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.0 GiB
1.3 TiB 61.28 1.00 87 up osd.175
182 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.3 GiB
1.4 TiB 59.78 0.98 85 up osd.182
185 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.51 1.02 89 up osd.185
187 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 59.70 0.98 85 up osd.187
188 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.2 GiB
1.4 TiB 59.70 0.98 85 up osd.188
192 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.61 1.03 89 up osd.192
195 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.89 0.98 85 up osd.195
198 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.4 GiB
1.3 TiB 62.56 1.03 89 up osd.198
200 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.89 1.03 89 up osd.200
204 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 5.5 GiB
1.3 TiB 60.26 0.99 85 up osd.204
205 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.72 1.03 89 up osd.205
206 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.71 1.03 89 up osd.206
211 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.5 GiB
1.4 TiB 59.79 0.98 85 up osd.211
-12 318.94116 - 322 TiB 196 TiB 195 TiB 2.8 GiB 585 GiB
126 TiB 60.80 1.00 - rack <rack>
-23 81.43176 - 81 TiB 50 TiB 50 TiB 737 MiB 150 GiB
32 TiB 61.24 1.00 - host <host>
142 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 31 MiB 6.5 GiB
1.2 TiB 63.96 1.05 91 up osd.142
145 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.1 GiB
1.3 TiB 60.73 1.00 86 up osd.145
147 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.97 0.98 85 up osd.147
148 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.77 1.03 89 up osd.148
153 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.84 0.98 85 up osd.153
154 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.2 GiB
1.4 TiB 59.93 0.98 85 up osd.154
155 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 61.91 1.02 88 up osd.155
156 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.60 1.03 89 up osd.156
165 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.3 GiB
1.4 TiB 60.07 0.98 85 up osd.165
170 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.92 0.98 85 up osd.170
174 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.2 GiB
1.4 TiB 59.95 0.98 85 up osd.174
176 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.6 GiB
1.4 TiB 60.00 0.98 85 up osd.176
180 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 61.96 1.02 88 up osd.180
181 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.1 GiB
1.3 TiB 62.85 1.03 89 up osd.181
183 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 62.08 1.02 88 up osd.183
186 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.65 0.98 85 up osd.186
190 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.79 1.03 89 up osd.190
191 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.7 GiB
1.3 TiB 62.71 1.03 89 up osd.191
194 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 59.76 0.98 85 up osd.194
197 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.77 1.03 89 up osd.197
201 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.0 GiB
1.3 TiB 60.73 1.00 86 up osd.201
208 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.89 0.98 85 up osd.208
209 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.4 GiB
1.4 TiB 59.88 0.98 85 up osd.209
210 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.9 GiB
1.3 TiB 62.98 1.03 89 up osd.210
-31 78.03877 - 78 TiB 48 TiB 48 TiB 710 MiB 146 GiB
30 TiB 61.20 1.00 - host <host>
220 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 59.80 0.98 85 up osd.220
221 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.76 0.98 85 up osd.221
222 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 61.92 1.02 88 up osd.222
224 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.67 1.03 89 up osd.224
230 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.8 GiB
1.4 TiB 59.79 0.98 85 up osd.230
231 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.3 GiB
1.3 TiB 62.72 1.03 89 up osd.231
233 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.87 0.98 85 up osd.233
236 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.3 GiB
1.3 TiB 61.97 1.02 88 up osd.236
241 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.75 1.03 90 up osd.241
246 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.6 GiB
1.3 TiB 62.79 1.03 89 up osd.246
247 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.62 1.03 89 up osd.247
248 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.59 1.03 89 up osd.248
253 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.5 GiB
1.4 TiB 59.08 0.97 84 up osd.253
254 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.76 1.03 89 up osd.254
257 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.6 GiB
1.4 TiB 59.22 0.97 84 up osd.257
260 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.0 GiB
1.4 TiB 59.12 0.97 84 up osd.260
265 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.3 GiB
1.3 TiB 62.84 1.03 89 up osd.265
266 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.60 1.03 89 up osd.266
267 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 59.13 0.97 84 up osd.267
268 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 33 MiB 6.6 GiB
1.3 TiB 62.56 1.03 89 up osd.268
277 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.4 GiB
1.4 TiB 59.19 0.97 84 up osd.277
279 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 59.12 0.97 84 up osd.279
282 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.4 GiB
1.3 TiB 62.81 1.03 89 up osd.282
-11 81.43176 - 81 TiB 50 TiB 50 TiB 748 MiB 142 GiB
31 TiB 61.83 1.01 - host <host>
41 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.8 GiB
1.4 TiB 59.94 0.98 85 up osd.41
42 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 5.8 GiB
1.3 TiB 61.03 1.00 86 up osd.42
44 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.8 GiB
1.4 TiB 60.16 0.99 85 up osd.44
46 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.2 GiB
1.2 TiB 63.55 1.04 90 up osd.46
47 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.6 GiB
1.4 TiB 60.09 0.99 85 up osd.47
48 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 5.7 GiB
1.2 TiB 63.65 1.04 90 up osd.48
49 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.2 GiB
1.2 TiB 63.67 1.04 90 up osd.49
50 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.2 GiB
1.2 TiB 63.56 1.04 90 up osd.50
51 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 60.01 0.98 85 up osd.51
52 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.8 GiB
1.4 TiB 59.98 0.98 85 up osd.52
53 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.7 GiB
1.4 TiB 60.08 0.98 85 up osd.53
54 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.2 GiB
1.3 TiB 62.83 1.03 89 up osd.54
55 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 5.9 GiB
1.3 TiB 62.11 1.02 88 up osd.55
56 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 5.9 GiB
1.3 TiB 62.80 1.03 89 up osd.56
57 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 33 MiB 6.2 GiB
1.2 TiB 63.78 1.05 90 up osd.57
58 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.1 GiB
1.2 TiB 63.76 1.05 90 up osd.58
59 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.3 GiB
1.2 TiB 63.65 1.04 90 up osd.59
60 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.7 GiB
1.4 TiB 60.20 0.99 85 up osd.60
61 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.7 GiB
1.4 TiB 60.14 0.99 85 up osd.61
62 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 60.07 0.98 85 up osd.62
63 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 5.5 GiB
1.4 TiB 60.01 0.98 85 up osd.63
65 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.7 GiB
1.3 TiB 62.28 1.02 88 up osd.65
66 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 33 MiB 6.5 GiB
1.3 TiB 63.04 1.03 89 up osd.66
136 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 5.7 GiB
1.2 TiB 63.41 1.04 90 up osd.136
-33 0.00009 - 3.4 TiB 28 GiB 26 GiB 6.3 MiB 2.0 GiB
3.4 TiB 0.81 0.01 - host <host-added-later>
283 nvme 0.00009 1.00000 3.4 TiB 28 GiB 26 GiB 6.3 MiB 2.0 GiB
3.4 TiB 0.81 0.01 1 up osd.283
-17 78.03877 - 78 TiB 48 TiB 48 TiB 712 MiB 145 GiB
30 TiB 61.49 1.01 - host <host>
94 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 62.70 1.03 89 up osd.94
97 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.2 TiB 63.32 1.04 90 up osd.97
100 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.0 GiB
1.4 TiB 59.71 0.98 85 up osd.100
104 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.01 1.02 88 up osd.104
105 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 61.94 1.02 88 up osd.105
106 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.4 GiB
1.2 TiB 63.57 1.04 90 up osd.106
107 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.9 GiB
1.4 TiB 59.77 0.98 85 up osd.107
108 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 33 MiB 6.5 GiB
1.2 TiB 63.50 1.04 90 up osd.108
109 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.2 GiB
1.4 TiB 59.96 0.98 85 up osd.109
110 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.2 GiB
1.2 TiB 63.45 1.04 90 up osd.110
111 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.0 TiB 31 MiB 6.2 GiB
1.3 TiB 60.51 0.99 86 up osd.111
112 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 61.23 1.00 87 up osd.112
113 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.4 GiB
1.4 TiB 59.94 0.98 85 up osd.113
114 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 61.21 1.00 87 up osd.114
115 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.22 1.04 90 up osd.115
116 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.9 GiB
1.4 TiB 59.87 0.98 85 up osd.116
117 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.72 0.98 85 up osd.117
118 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.3 GiB
1.3 TiB 62.67 1.03 89 up osd.118
119 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.0 TiB 30 MiB 6.3 GiB
1.3 TiB 60.45 0.99 86 up osd.119
120 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.3 GiB
1.3 TiB 62.82 1.03 89 up osd.120
121 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.2 GiB
1.4 TiB 59.93 0.98 85 up osd.121
139 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 5.6 GiB
1.4 TiB 60.05 0.98 85 up osd.139
140 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.63 1.03 89 up osd.140
-4 322.33405 - 322 TiB 196 TiB 195 TiB 2.8 GiB 577 GiB
126 TiB 60.80 1.00 - rack <rack>
-15 78.03877 - 78 TiB 48 TiB 48 TiB 707 MiB 144 GiB
30 TiB 61.36 1.01 - host <host>
70 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 5.8 GiB
1.2 TiB 63.59 1.04 90 up osd.70
72 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.4 GiB
1.2 TiB 63.40 1.04 90 up osd.72
73 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.7 GiB
1.4 TiB 57.85 0.95 82 up osd.73
74 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.7 GiB
1.2 TiB 63.41 1.04 90 up osd.74
75 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.5 GiB
1.2 TiB 63.43 1.04 90 up osd.75
76 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.3 GiB
1.2 TiB 63.44 1.04 90 up osd.76
77 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 61.26 1.00 87 up osd.77
78 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.60 1.03 89 up osd.78
79 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 57.68 0.95 82 up osd.79
80 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 57.87 0.95 82 up osd.80
81 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.2 GiB
1.3 TiB 61.37 1.01 87 up osd.81
82 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.73 1.03 89 up osd.82
83 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.8 GiB
1.2 TiB 63.42 1.04 90 up osd.83
84 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 57.77 0.95 82 up osd.84
85 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.2 GiB
1.3 TiB 62.64 1.03 89 up osd.85
86 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 31 MiB 6.2 GiB
1.2 TiB 63.41 1.04 90 up osd.86
87 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.1 GiB
1.3 TiB 61.10 1.00 87 up osd.87
88 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.72 1.03 89 up osd.88
89 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.4 GiB
1.4 TiB 58.05 0.95 82 up osd.89
90 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.8 GiB
1.4 TiB 57.74 0.95 82 up osd.90
91 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.3 GiB
1.3 TiB 60.61 0.99 86 up osd.91
92 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 62.60 1.03 89 up osd.92
93 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.7 GiB
1.3 TiB 62.50 1.02 89 up osd.93
-3 81.43176 - 81 TiB 50 TiB 50 TiB 755 MiB 135 GiB
31 TiB 61.42 1.01 - host <host>
0 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.3 GiB
1.4 TiB 58.59 0.96 83 up osd.0
1 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 6.0 GiB
1.2 TiB 63.66 1.04 90 up osd.1
2 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.6 GiB
1.3 TiB 62.16 1.02 88 up osd.2
3 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.4 GiB
1.4 TiB 58.63 0.96 83 up osd.3
5 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.3 GiB
1.4 TiB 58.91 0.97 83 up osd.5
6 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.6 GiB
1.3 TiB 62.01 1.02 88 up osd.6
7 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.4 GiB
1.2 TiB 63.54 1.04 90 up osd.7
8 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.2 GiB
1.4 TiB 58.65 0.96 83 up osd.8
9 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.6 GiB
1.3 TiB 62.89 1.03 89 up osd.9
10 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.3 GiB
1.4 TiB 58.76 0.96 83 up osd.10
11 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 32 MiB 5.6 GiB
1.2 TiB 63.59 1.04 90 up osd.11
12 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 34 MiB 5.9 GiB
1.2 TiB 63.71 1.04 90 up osd.12
13 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.7 GiB
1.3 TiB 60.97 1.00 86 up osd.13
14 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 5.7 GiB
1.3 TiB 60.78 1.00 86 up osd.14
15 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 5.9 GiB
1.2 TiB 63.45 1.04 90 up osd.15
16 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 5.8 GiB
1.2 TiB 63.51 1.04 90 up osd.16
18 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.1 GiB
1.4 TiB 58.03 0.95 82 up osd.18
19 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 33 MiB 5.6 GiB
1.3 TiB 62.93 1.03 89 up osd.19
20 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.7 GiB
1.3 TiB 62.74 1.03 89 up osd.20
21 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.0 GiB
1.2 TiB 63.53 1.04 90 up osd.21
22 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.2 TiB 33 MiB 6.0 GiB
1.2 TiB 63.73 1.04 90 up osd.22
67 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.3 GiB
1.4 TiB 57.97 0.95 82 up osd.67
71 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 5.9 GiB
1.2 TiB 63.51 1.04 90 up osd.71
278 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 5.3 GiB
1.4 TiB 57.85 0.95 82 up osd.278
-27 81.43176 - 81 TiB 49 TiB 49 TiB 720 MiB 148 GiB
33 TiB 59.74 0.98 - host <host>
213 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 1.9 TiB 29 MiB 6.2 GiB
1.4 TiB 57.64 0.95 82 up osd.213
214 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.8 GiB
1.4 TiB 57.75 0.95 82 up osd.214
215 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.76 1.03 89 up osd.215
216 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.6 GiB
1.4 TiB 57.64 0.95 82 up osd.216
225 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.66 1.03 89 up osd.225
226 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.70 1.03 89 up osd.226
227 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.2 GiB
1.4 TiB 57.72 0.95 82 up osd.227
228 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 62.65 1.03 89 up osd.228
237 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.0 TiB 31 MiB 6.2 GiB
1.3 TiB 60.53 0.99 86 up osd.237
238 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.6 GiB
1.4 TiB 57.68 0.95 82 up osd.238
239 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.2 GiB
1.3 TiB 61.27 1.00 87 up osd.239
240 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.5 GiB
1.3 TiB 61.88 1.01 88 up osd.240
249 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 58.63 0.96 83 up osd.249
250 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.4 GiB
1.3 TiB 61.86 1.01 88 up osd.250
251 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.0 GiB
1.4 TiB 58.01 0.95 82 up osd.251
252 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.4 GiB
1.3 TiB 62.61 1.03 89 up osd.252
261 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 57.71 0.95 82 up osd.261
262 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.4 GiB
1.3 TiB 62.60 1.03 89 up osd.262
263 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.0 GiB
1.4 TiB 57.84 0.95 82 up osd.263
264 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 28 MiB 6.1 GiB
1.4 TiB 57.81 0.95 82 up osd.264
273 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 1.9 TiB 30 MiB 6.4 GiB
1.4 TiB 57.57 0.94 82 up osd.273
274 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 57.77 0.95 82 up osd.274
275 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 5.7 GiB
1.3 TiB 60.69 0.99 86 up osd.275
276 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.8 GiB
1.4 TiB 57.89 0.95 82 up osd.276
-21 81.43176 - 81 TiB 49 TiB 49 TiB 731 MiB 150 GiB
32 TiB 60.71 1.00 - host <host>
64 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 5.8 GiB
1.3 TiB 62.98 1.03 89 up osd.64
141 nvme 3.39299 1.00000 3.4 TiB 2.2 TiB 2.1 TiB 32 MiB 6.4 GiB
1.2 TiB 63.52 1.04 90 up osd.141
146 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.1 GiB
1.3 TiB 62.56 1.03 89 up osd.146
150 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 57.69 0.95 82 up osd.150
151 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.75 1.03 89 up osd.151
157 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.3 GiB
1.3 TiB 62.62 1.03 89 up osd.157
158 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 58.55 0.96 83 up osd.158
163 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 57.71 0.95 82 up osd.163
164 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.6 GiB
1.3 TiB 62.83 1.03 89 up osd.164
166 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 62.81 1.03 89 up osd.166
167 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.84 1.03 90 up osd.167
171 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.1 GiB
1.4 TiB 57.85 0.95 82 up osd.171
173 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.1 GiB
1.3 TiB 62.76 1.03 89 up osd.173
177 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 31 MiB 6.4 GiB
1.4 TiB 59.20 0.97 84 up osd.177
178 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.0 TiB 30 MiB 6.6 GiB
1.3 TiB 60.48 0.99 86 up osd.178
179 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 6.2 GiB
1.4 TiB 57.68 0.95 82 up osd.179
184 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.1 GiB
1.4 TiB 59.26 0.97 84 up osd.184
189 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 30 MiB 6.2 GiB
1.3 TiB 61.33 1.01 87 up osd.189
193 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.4 GiB
1.3 TiB 62.78 1.03 89 up osd.193
196 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 32 MiB 6.7 GiB
1.3 TiB 62.71 1.03 89 up osd.196
199 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.9 GiB
1.4 TiB 57.84 0.95 82 up osd.199
202 nvme 3.39299 1.00000 3.4 TiB 2.1 TiB 2.1 TiB 31 MiB 6.5 GiB
1.3 TiB 62.62 1.03 89 up osd.202
207 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 30 MiB 6.3 GiB
1.4 TiB 57.90 0.95 82 up osd.207
212 nvme 3.39299 1.00000 3.4 TiB 2.0 TiB 2.0 TiB 29 MiB 5.8 GiB
1.4 TiB 57.73 0.95 82 up osd.212
TOTAL 964 TiB 588 TiB 586 TiB 8.5 GiB 1.7 TiB
376 TiB 60.99
MIN/MAX VAR: 0.01/1.05 STDDEV: 4.05
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
