Hi Ceph Admins,
This night our ceph cluster got all pools 100% full. This happend after
osd.56 (95% used) reached OSD_FULL state.
ceph versions 12.2.2
Logs
2018-03-03 17:15:22.560710 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224452
: cluster [ERR] overall HEALTH_ERR noscrub,nodeep-scrub flag(s) set; 1
backfillfull osd(s); 5 nearfull osd(s); 21 pool(s) backfillfull;
638551/287271738 objects misplaced (0.222%); Degraded data redundancy:
253066/287271738 objects degraded (0.088%), 25 pgs unclean; Degraded data
redundancy (low space): 25 pgs backfill_toofull
2018-03-03 17:15:42.513194 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224515
: cluster [WRN] Health check update: 638576/287284518 objects misplaced
(0.222%) (OBJECT_MISPLACED)
2018-03-03 17:15:42.513256 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224516
: cluster [WRN] Health check update: Degraded data redundancy:
253266/287284518 objects degraded (0.088%), 25 pgs unclean (PG_DEGRADED)
2018-03-03 17:15:44.684928 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224524
: cluster [ERR] Health check failed: 1 full osd(s) (OSD_FULL)
2018-03-03 17:15:44.684969 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224525
: cluster [WRN] Health check failed: 21 pool(s) full (POOL_FULL)
2018-03-03 17:15:44.684987 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224526
: cluster [INF] Health check cleared: OSD_BACKFILLFULL (was: 1 backfillfull
osd(s))
2018-03-03 17:15:44.685013 mon.cephnode01 mon.0 10.212.32.18:6789/0 5224527
: cluster [INF] Health check cleared: POOL_BACKFILLFULL (was: 21 pool(s)
backfillfull)
# ceph df detail from crush time
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
381T 102T 278T 73.05 38035k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
USED %USED MAX AVAIL OBJECTS DIRTY READ
WRITE RAW USED
rbd 0 N/A N/A
0 0 0 0 0 1
134k 0
vms 1 N/A N/A
0 0 0 0 0 0
0 0
images 2 N/A N/A
7659M 100.00 0 1022 1022 110k
5668 22977M
volumes 3 N/A N/A
40991G 100.00 0 10514980 10268k 3404M
4087M 120T
.rgw.root 4 N/A N/A
1588 100.00 0 4 4 402k
4 4764
default.rgw.control 5 N/A N/A
0 0 0 8 8 0
0 0
default.rgw.data.root 6 N/A N/A
94942 100.00 0 339 339 257k
6422 278k
default.rgw.gc 7 N/A N/A
0 0 0 32 32 3125M
7410k 0
default.rgw.log 8 N/A N/A
0 0 0 186 186 27222k
18146k 0
default.rgw.users.uid 9 N/A N/A
4252 100.00 0 17 17 262k
64561 12756
default.rgw.usage 10 N/A N/A
0 0 0 8 8 332k
665k 0
default.rgw.users.email 11 N/A N/A
87 100.00 0 4 4 0
4 261
default.rgw.users.keys 12 N/A N/A
206 100.00 0 11 11 459
23 618
default.rgw.users.swift 13 N/A N/A
40 100.00 0 3 3 0
3 120
default.rgw.buckets.index 14 N/A N/A
0 0 0 210 210 321M
41709k 0
default.rgw.buckets.non-ec 16 N/A N/A
0 0 0 114 114 18006
12055 0
default.rgw.buckets.extra 17 N/A N/A
0 0 0 0 0 0
0 0
.rgw.buckets.extra 18 N/A N/A
0 0 0 0 0 0
0 0
default.rgw.buckets.data 20 N/A N/A
104T 100.00 0 28334451 27670k 160M
156M 156T
benchmark_replicated 21 N/A N/A
87136M 100.00 0 21792 21792 1450k
4497k 255G
benchmark_erasure_coded 22 N/A N/A
292G 100.00 0 74779 74779 61288
680k 439G
#
What we did to reclaim some space is:
- deleted two benchmark pools
- reweight full osd.56 from 1.0 to 0.85
- added new node - cephnode10 (cluster has grown from 9 to 10 nodes but I
had to do crush reweight down to 0 on new OSDs as a lot of slow requestes
(like 3000+) occured and customer IOPS went totally down. Adding one OSD at
a time now)
Current status
# ceph -s
cluster:
id: 1023c49f-3a10-42de-9f62-9b122db32f1f
health: HEALTH_ERR
noscrub,nodeep-scrub flag(s) set
5 nearfull osd(s)
19 pool(s) nearfull
16151257/286563963 objects misplaced (5.636%)
Degraded data redundancy: 20949/286563963 objects degraded
(0.007%), 431 pgs unclean, 28 pgs degraded, 1 pg undersized
Degraded data redundancy (low space): 15 pgs backfill_toofull
services:
mon: 3 daemons, quorum cephnode01,cephnode02,cephnode03
mgr: cephnode02(active), standbys: cephnode03, cephnode01
osd: 120 osds: 117 up, 117 in; 405 remapped pgs
flags noscrub,nodeep-scrub
rgw: 3 daemons active
data:
pools: 19 pools, 3760 pgs
objects: 37941k objects, 144 TB
usage: 278 TB used, 146 TB / 425 TB avail
pgs: 20949/286563963 objects degraded (0.007%)
16151257/286563963 objects misplaced (5.636%)
3329 active+clean
370 active+remapped+backfill_wait
26 active+recovery_wait+degraded
18 active+remapped+backfilling
15 active+remapped+backfill_wait+backfill_toofull
1 active+recovery_wait+degraded+remapped
1 active+undersized+degraded+remapped+backfilling
io:
client: 18337 B/s rd, 29269 kB/s wr, 1 op/s rd, 234 op/s wr
recovery: 946 MB/s, 243 objects/s
#
# ceph df detail
GLOBAL:
SIZE AVAIL RAW USED %RAW USED OBJECTS
425T 146T 278T 65.50 37941k
POOLS:
NAME ID QUOTA OBJECTS QUOTA BYTES
USED %USED MAX AVAIL OBJECTS DIRTY READ
WRITE RAW USED
rbd 0 N/A N/A
0 0 7415G 0 0 1
134k 0
vms 1 N/A N/A
0 0 7415G 0 0 0
0 0
images 2 N/A N/A
7659M 0.10 7415G 1022 1022 110k
5668 22445M
volumes 3 N/A N/A
40992G 84.68 7415G 10515231 10268k 3416M
4090M 120T
.rgw.root 4 N/A N/A
1588 0 7415G 4 4 141k
4 4764
default.rgw.control 5 N/A N/A
0 0 7415G 8 8 0
0 0
default.rgw.data.root 6 N/A N/A
94942 0 7415G 339 339 257k
6422 278k
default.rgw.gc 7 N/A N/A
0 0 7415G 32 32 3125M
7430k 0
default.rgw.log 8 N/A N/A
0 0 7415G 186 186 27249k
18164k 0
default.rgw.users.uid 9 N/A N/A
4252 0 7415G 17 17 263k
64577 12756
default.rgw.usage 10 N/A N/A
0 0 7415G 8 8 332k
665k 0
default.rgw.users.email 11 N/A N/A
87 0 7415G 4 4 0
4 261
default.rgw.users.keys 12 N/A N/A
206 0 7415G 11 11 483
23 580
default.rgw.users.swift 13 N/A N/A
40 0 7415G 3 3 0
3 120
default.rgw.buckets.index 14 N/A N/A
0 0 7415G 210 210 321M
41709k 0
default.rgw.buckets.non-ec 16 N/A N/A
0 0 7415G 114 114 18006
12055 0
default.rgw.buckets.extra 17 N/A N/A
0 0 7415G 0 0 0
0 0
.rgw.buckets.extra 18 N/A N/A
0 0 7415G 0 0 0
0 0
default.rgw.buckets.data 20 N/A N/A
104T 87.85 14831G 28334711 27670k 160M
156M 157T
#
Most utilized pools are: volumes (replicated pool) and
default.rgw.buckets.data (EC pool, k=6,m=3)
pool 3 'volumes' replicated size 3 min_size 2 crush_rule 0 object_hash
rjenkins pg_num 1024 pgp_num 1024 last_change 10047 flags
hashpspool,backfillfull stripe_width 0 application rbd
removed_snaps [1~3]
pool 20 'default.rgw.buckets.data' erasure size 9 min_size 6 crush_rule 1
object_hash rjenkins pg_num 1024 pgp_num 1024 last_change 10047 flags
hashpspool,backfillfull stripe_width 4224 application rgw
Crush rules for above pools:
# rules
rule replicated_ruleset {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type rack # !!! rack as failure domain
step emit
}
rule ec_rule_k6_m3 {
id 1
type erasure
min_size 3
max_size 9
step set_chooseleaf_tries 5
step set_choose_tries 100
step take default
step chooseleaf indep 0 type host # !!! host as failure domain
step emit
}
And finally cluster topology
# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME
-1 392.72797 - 425T 278T 146T 65.51 1.00 - root
default
-6 392.72797 - 425T 278T 146T 65.51 1.00 - region
region01
-5 392.72797 - 425T 278T 146T 65.51 1.00 -
datacenter dc01
-4 392.72797 - 425T 278T 146T 65.51 1.00 -
room room01
-8 43.63699 - 44684G 31703G 12980G 70.95 1.08 -
rack rack01
-7 43.63699 - 44684G 31703G 12980G 70.95 1.08 -
host cephnode01
0 hdd 3.63599 1.00000 3723G 2957G 765G 79.43 1.21 178
osd.0
2 hdd 3.63599 1.00000 3723G 2407G 1315G 64.66 0.99 157
osd.2
4 hdd 3.63599 1.00000 3723G 2980G 742G 80.05 1.22 184
osd.4
6 hdd 3.63599 1.00000 3723G 2768G 955G 74.34 1.13 170
osd.6
8 hdd 3.63599 1.00000 3723G 2704G 1019G 72.62 1.11 172
osd.8
11 hdd 3.63599 1.00000 3723G 2899G 824G 77.87 1.19 181
osd.11
12 hdd 3.63599 1.00000 3723G 2788G 935G 74.89 1.14 183
osd.12
14 hdd 3.63599 1.00000 3723G 2139G 1584G 57.44 0.88 139
osd.14
16 hdd 3.63599 1.00000 3723G 2672G 1050G 71.78 1.10 174
osd.16
18 hdd 3.63599 1.00000 3723G 2575G 1148G 69.17 1.06 166
osd.18
20 hdd 3.63599 1.00000 3723G 2395G 1328G 64.33 0.98 149
osd.20
22 hdd 3.63599 1.00000 3723G 2414G 1309G 64.83 0.99 161
osd.22
-3 43.63699 - 44684G 32329G 12354G 72.35 1.10 -
rack rack02
-2 43.63699 - 44684G 32329G 12354G 72.35 1.10 -
host cephnode02
1 hdd 3.63599 1.00000 3723G 2874G 848G 77.21 1.18 172
osd.1
3 hdd 3.63599 1.00000 3723G 3287G 436G 88.27 1.35 190
osd.3
5 hdd 3.63599 1.00000 3723G 2588G 1135G 69.50 1.06 151
osd.5
7 hdd 3.63599 1.00000 3723G 2566G 1156G 68.94 1.05 156
osd.7
9 hdd 3.63599 1.00000 3723G 2481G 1242G 66.65 1.02 164
osd.9
10 hdd 3.63599 1.00000 3723G 2622G 1101G 70.43 1.08 156
osd.10
13 hdd 3.63599 1.00000 3723G 2498G 1225G 67.08 1.02 150
osd.13
15 hdd 3.63599 1.00000 3723G 2664G 1058G 71.56 1.09 167
osd.15
17 hdd 3.63599 1.00000 3723G 2510G 1213G 67.42 1.03 163
osd.17
19 hdd 3.63599 1.00000 3723G 2562G 1161G 68.82 1.05 162
osd.19
21 hdd 3.63599 1.00000 3723G 2683G 1040G 72.05 1.10 169
osd.21
23 hdd 3.63599 1.00000 3723G 2989G 734G 80.28 1.23 169
osd.23
-10 43.63699 - 44684G 32556G 12128G 72.86 1.11 -
rack rack03
-9 43.63699 - 44684G 32556G 12128G 72.86 1.11 -
host cephnode03
24 hdd 3.63599 1.00000 3723G 2757G 966G 74.05 1.13 155
osd.24
25 hdd 3.63599 1.00000 3723G 3003G 720G 80.66 1.23 186
osd.25
26 hdd 3.63599 1.00000 3723G 2494G 1229G 66.98 1.02 168
osd.26
28 hdd 3.63599 1.00000 3723G 3021G 701G 81.15 1.24 180
osd.28
30 hdd 3.63599 1.00000 3723G 2554G 1169G 68.60 1.05 164
osd.30
32 hdd 3.63599 1.00000 3723G 2060G 1662G 55.34 0.84 147
osd.32
34 hdd 3.63599 1.00000 3723G 3131G 592G 84.08 1.28 181
osd.34
36 hdd 3.63599 1.00000 3723G 2512G 1211G 67.47 1.03 162
osd.36
38 hdd 3.63599 1.00000 3723G 2408G 1315G 64.68 0.99 157
osd.38
40 hdd 3.63599 1.00000 3723G 2997G 726G 80.49 1.23 194
osd.40
42 hdd 3.63599 1.00000 3723G 2645G 1078G 71.05 1.08 161
osd.42
44 hdd 3.63599 1.00000 3723G 2969G 754G 79.74 1.22 173
osd.44
-12 43.63699 - 44684G 32504G 12179G 72.74 1.11 -
rack rack04
-11 43.63699 - 44684G 32504G 12179G 72.74 1.11 -
host cephnode04
27 hdd 3.63599 1.00000 3723G 2947G 775G 79.16 1.21 186
osd.27
29 hdd 3.63599 1.00000 3723G 3095G 628G 83.13 1.27 175
osd.29
31 hdd 3.63599 1.00000 3723G 2514G 1209G 67.52 1.03 163
osd.31
33 hdd 3.63599 1.00000 3723G 2557G 1166G 68.68 1.05 160
osd.33
35 hdd 3.63599 1.00000 3723G 3215G 508G 86.35 1.32 183
osd.35
37 hdd 3.63599 1.00000 3723G 2455G 1268G 65.93 1.01 151
osd.37
39 hdd 3.63599 1.00000 3723G 2335G 1387G 62.73 0.96 155
osd.39
41 hdd 3.63599 1.00000 3723G 2774G 949G 74.51 1.14 165
osd.41
43 hdd 3.63599 1.00000 3723G 2764G 959G 74.24 1.13 169
osd.43
45 hdd 3.63599 1.00000 3723G 2553G 1169G 68.59 1.05 163
osd.45
46 hdd 3.63599 1.00000 3723G 2645G 1077G 71.06 1.08 167
osd.46
47 hdd 3.63599 1.00000 3723G 2644G 1079G 71.02 1.08 156
osd.47
-14 39.99585 - 33513G 27770G 5742G 82.86 1.26 -
rack rack05
-13 39.99585 - 33513G 27770G 5742G 82.86 1.26 -
host cephnode05
48 hdd 3.63599 0.90002 3723G 3310G 413G 88.89 1.36 211
osd.48
49 hdd 3.63599 0.80005 3723G 3029G 694G 81.36 1.24 182
osd.49
50 hdd 3.63599 0.85004 3723G 2918G 804G 78.38 1.20 167
osd.50
51 hdd 3.63599 0.85004 3723G 3103G 620G 83.33 1.27 186
osd.51
52 hdd 0 0 0 0 0 0 0 0
osd.52
53 hdd 3.63599 0 0 0 0 0 0 0
osd.53
54 hdd 3.63599 0 0 0 0 0 0 0
osd.54
55 hdd 3.63599 0.85004 3723G 3003G 720G 80.65 1.23 178
osd.55
56 hdd 3.63599 0.84999 3723G 3347G 376G 89.89 1.37 189
osd.56
57 hdd 3.63599 0.75006 3723G 2707G 1016G 72.71 1.11 161
osd.57
58 hdd 3.63599 0.80005 3723G 3228G 495G 86.71 1.32 186
osd.58
59 hdd 3.63599 0.80005 3723G 3122G 601G 83.85 1.28 194
osd.59
-16 43.63699 - 44684G 33402G 11281G 74.75 1.14 -
rack rack06
-15 43.63699 - 44684G 33402G 11281G 74.75 1.14 -
host cephnode06
60 hdd 3.63599 1.00000 3723G 2317G 1406G 62.22 0.95 149
osd.60
61 hdd 3.63599 1.00000 3723G 3039G 684G 81.62 1.25 183
osd.61
62 hdd 3.63599 1.00000 3723G 2945G 778G 79.09 1.21 189
osd.62
63 hdd 3.63599 1.00000 3723G 2923G 800G 78.50 1.20 166
osd.63
64 hdd 3.63599 1.00000 3723G 3057G 665G 82.11 1.25 180
osd.64
65 hdd 3.63599 1.00000 3723G 2989G 733G 80.30 1.23 170
osd.65
66 hdd 3.63599 1.00000 3723G 2764G 959G 74.25 1.13 166
osd.66
67 hdd 3.63599 1.00000 3723G 2811G 912G 75.50 1.15 175
osd.67
68 hdd 3.63599 1.00000 3723G 1785G 1938G 47.95 0.73 139
osd.68
69 hdd 3.63599 1.00000 3723G 2744G 979G 73.69 1.12 159
osd.69
70 hdd 3.63599 1.00000 3723G 3068G 655G 82.40 1.26 178
osd.70
71 hdd 3.63599 1.00000 3723G 2956G 767G 79.40 1.21 174
osd.71
-18 43.63699 - 44684G 33524G 11159G 75.03 1.15 -
rack rack07
-17 43.63699 - 44684G 33524G 11159G 75.03 1.15 -
host cephnode07
72 hdd 3.63599 1.00000 3723G 2901G 822G 77.91 1.19 178
osd.72
73 hdd 3.63599 1.00000 3723G 2612G 1110G 70.16 1.07 168
osd.73
74 hdd 3.63599 1.00000 3723G 2870G 853G 77.09 1.18 172
osd.74
75 hdd 3.63599 1.00000 3723G 2813G 910G 75.56 1.15 169
osd.75
76 hdd 3.63599 1.00000 3723G 2861G 862G 76.85 1.17 170
osd.76
77 hdd 3.63599 1.00000 3723G 2807G 916G 75.39 1.15 168
osd.77
78 hdd 3.63599 1.00000 3723G 2678G 1045G 71.92 1.10 156
osd.78
79 hdd 3.63599 1.00000 3723G 2556G 1166G 68.67 1.05 160
osd.79
80 hdd 3.63599 1.00000 3723G 3082G 640G 82.79 1.26 190
osd.80
81 hdd 3.63599 1.00000 3723G 2418G 1305G 64.94 0.99 144
osd.81
82 hdd 3.63599 1.00000 3723G 2881G 841G 77.39 1.18 161
osd.82
83 hdd 3.63599 1.00000 3723G 3039G 683G 81.64 1.25 175
osd.83
-20 90.91017 - 130T 61630G 72421G 45.98 0.70 -
rack rack08
-19 43.63699 - 44684G 30861G 13823G 69.06 1.05 -
host cephnode08
84 hdd 3.63599 1.00000 3723G 2532G 1190G 68.02 1.04 157
osd.84
85 hdd 3.63599 1.00000 3723G 2518G 1205G 67.64 1.03 166
osd.85
86 hdd 3.63599 1.00000 3723G 2504G 1219G 67.25 1.03 151
osd.86
87 hdd 3.63599 1.00000 3723G 2698G 1024G 72.47 1.11 161
osd.87
88 hdd 3.63599 1.00000 3723G 2527G 1196G 67.87 1.04 147
osd.88
89 hdd 3.63599 1.00000 3723G 2508G 1215G 67.36 1.03 142
osd.89
90 hdd 3.63599 1.00000 3723G 2317G 1406G 62.24 0.95 142
osd.90
91 hdd 3.63599 1.00000 3723G 2582G 1140G 69.36 1.06 147
osd.91
92 hdd 3.63599 1.00000 3723G 2656G 1066G 71.35 1.09 144
osd.92
93 hdd 3.63599 1.00000 3723G 2448G 1275G 65.74 1.00 154
osd.93
94 hdd 3.63599 1.00000 3723G 2783G 939G 74.76 1.14 163
osd.94
95 hdd 3.63599 1.00000 3723G 2782G 941G 74.73 1.14 152
osd.95
-21 43.63678 - 44684G 30331G 14353G 67.88 1.04 -
host cephnode09
96 hdd 3.63640 1.00000 3723G 3003G 719G 80.67 1.23 161
osd.96
97 hdd 3.63640 1.00000 3723G 2581G 1142G 69.32 1.06 151
osd.97
98 hdd 3.63640 1.00000 3723G 2118G 1605G 56.88 0.87 140
osd.98
99 hdd 3.63640 1.00000 3723G 2926G 796G 78.60 1.20 165
osd.99
100 hdd 3.63640 1.00000 3723G 2492G 1231G 66.92 1.02 149
osd.100
101 hdd 3.63640 1.00000 3723G 2605G 1117G 69.98 1.07 165
osd.101
102 hdd 3.63640 1.00000 3723G 2159G 1563G 58.01 0.89 141
osd.102
103 hdd 3.63640 1.00000 3723G 2328G 1395G 62.53 0.95 146
osd.103
104 hdd 3.63640 1.00000 3723G 2624G 1099G 70.48 1.08 163
osd.104
105 hdd 3.63640 1.00000 3723G 2582G 1141G 69.34 1.06 142
osd.105
106 hdd 3.63640 1.00000 3723G 2401G 1322G 64.48 0.98 161
osd.106
107 hdd 3.63640 1.00000 3723G 2507G 1216G 67.33 1.03 159
osd.107
-43 3.63640 - 44684G 438G 44245G 0.98 0.01 -
host cephnode10 ## Added after cluster pools got full
108 hdd 3.63640 1.00000 3723G 51915M 3672G 1.36 0.02 36
osd.108
109 hdd 0 1.00000 3723G 72735M 3652G 1.91 0.03 4
osd.109
110 hdd 0 1.00000 3723G 36948M 3687G 0.97 0.01 2
osd.110
111 hdd 0 1.00000 3723G 37043M 3687G 0.97 0.01 2
osd.111
112 hdd 0 1.00000 3723G 72382M 3652G 1.90 0.03 4
osd.112
113 hdd 0 1.00000 3723G 54850M 3670G 1.44 0.02 3
osd.113
114 hdd 0 1.00000 3723G 36664M 3687G 0.96 0.01 2
osd.114
115 hdd 0 1.00000 3723G 36087M 3688G 0.95 0.01 2
osd.115
116 hdd 0 1.00000 3723G 12066M 3711G 0.32 0.00 0
osd.116
117 hdd 0 1.00000 3723G 36793M 3687G 0.96 0.01 2
osd.117
118 hdd 0 1.00000 3723G 775M 3722G 0.02 0 0
osd.118
119 hdd 0 1.00000 3723G 760M 3722G 0.02 0 0
osd.119
TOTAL 425T 278T 146T 65.51
MIN/MAX VAR: 0/1.37 STDDEV: 23.07
#
I'm wondering why one FULL OSD made all cluster pools full? Does OSD_FULL
state stop write operations to all OSDs on the node that full OSD resides
or just to concerned OSD ?
Should pg_num/pgp_num be increased to get better data balancing across all
OSDs?
Why there is only 7415G MAX AVAIL for volumes pool and 14831G for
default.rgw.buckets.data pool while cluster %RAW USED is 65.50 (only?) ? Is
it somehow related to bad looking node cephnode05 (OSDs highly utilized)
and the fact that K+M of EC pool was equal to the number of nodes in the
cluster?
Best Regards
Jakub
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
