Device classes are implemented with magic invisible crush trees; you've got two completely independent trees internally: one for crush rules mapping to HDDs, one to legacy crush rules not specifying a device class.
The balancer *should* be aware of this and ignore it, but I'm not sure about the state of the balancer on Luminous. There were quite a few problems in older versions, lots of them have been fixed in backports. The upmap balancer is much better than the crush-compat balancer, but it requires all clients to run Luminous or later. Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Wed, Jun 26, 2019 at 10:21 AM Wolfgang Lendl < wolfgang.le...@meduniwien.ac.at> wrote: > Hi, > > tried to enable the ceph balancer on a 12.2.12 cluster and got this: > > mgr[balancer] Some osds belong to multiple subtrees: [0, 1, 2, 3, 4, 5, 6, 7, > 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, > 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, > 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, > 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, > 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, > 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, > 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, > 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, > 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, > 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, > 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, > 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, > 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, > 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, > 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, > 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, > 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, > 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, > 298, 299, 300, 301, 302, 303, 304, 305] > > I'm not aware of any additional subtree - maybe someone can enlighten me: > > ceph balancer status > { > "active": true, > "plans": [], > "mode": "crush-compat" > } > > ceph osd crush tree > ID CLASS WEIGHT (compat) TYPE NAME > -1 3176.04785 root default > -7 316.52490 316.52490 host node0 > 0 hdd 9.09560 9.09560 osd.0 > 4 hdd 9.09560 9.09560 osd.4 > 8 hdd 9.09560 9.09560 osd.8 > 10 hdd 9.09560 9.09560 osd.10 > 12 hdd 9.09560 9.09560 osd.12 > 16 hdd 9.09560 9.09560 osd.16 > 20 hdd 9.09560 9.09560 osd.20 > 21 hdd 9.09560 9.09560 osd.21 > 26 hdd 9.09560 9.09560 osd.26 > 29 hdd 9.09560 9.09560 osd.29 > 31 hdd 9.09560 9.09560 osd.31 > 35 hdd 9.09560 9.09560 osd.35 > 37 hdd 9.09560 9.09560 osd.37 > 44 hdd 9.09560 9.09560 osd.44 > 47 hdd 9.09560 9.09560 osd.47 > 56 hdd 9.09560 9.09560 osd.56 > 59 hdd 9.09560 9.09560 osd.59 > 65 hdd 9.09560 9.09560 osd.65 > 71 hdd 9.09560 9.09560 osd.71 > 77 hdd 9.09560 9.09560 osd.77 > 80 hdd 9.09560 9.09560 osd.80 > 83 hdd 9.09569 9.09569 osd.83 > 86 hdd 9.09560 9.09560 osd.86 > 88 hdd 9.09560 9.09560 osd.88 > 94 hdd 10.91409 10.91409 osd.94 > 95 hdd 10.91409 10.91409 osd.95 > 98 hdd 10.91409 10.91409 osd.98 > 99 hdd 10.91409 10.91409 osd.99 > 238 hdd 9.09569 9.09569 osd.238 > 239 hdd 9.09569 9.09569 osd.239 > 240 hdd 9.09569 9.09569 osd.240 > 241 hdd 9.09569 9.09569 osd.241 > 242 hdd 9.09569 9.09569 osd.242 > 243 hdd 9.09569 9.09569 osd.243 > -3 316.52426 316.52426 host node1 > 1 hdd 9.09560 9.09560 osd.1 > 5 hdd 9.09560 9.09560 osd.5 > 6 hdd 9.09560 9.09560 osd.6 > 11 hdd 9.09560 9.09560 osd.11 > 13 hdd 9.09560 9.09560 osd.13 > 15 hdd 9.09560 9.09560 osd.15 > 19 hdd 9.09560 9.09560 osd.19 > 23 hdd 9.09560 9.09560 osd.23 > 25 hdd 9.09560 9.09560 osd.25 > 28 hdd 9.09560 9.09560 osd.28 > 32 hdd 9.09560 9.09560 osd.32 > 34 hdd 9.09560 9.09560 osd.34 > 38 hdd 9.09560 9.09560 osd.38 > 41 hdd 9.09560 9.09560 osd.41 > 43 hdd 9.09560 9.09560 osd.43 > 46 hdd 9.09560 9.09560 osd.46 > 49 hdd 9.09560 9.09560 osd.49 > 52 hdd 9.09560 9.09560 osd.52 > 55 hdd 9.09560 9.09560 osd.55 > 58 hdd 9.09560 9.09560 osd.58 > 61 hdd 9.09560 9.09560 osd.61 > 64 hdd 9.09560 9.09560 osd.64 > 67 hdd 9.09560 9.09560 osd.67 > 70 hdd 9.09560 9.09560 osd.70 > 73 hdd 9.09560 9.09560 osd.73 > 76 hdd 9.09560 9.09560 osd.76 > 79 hdd 9.09560 9.09560 osd.79 > 81 hdd 9.09560 9.09560 osd.81 > 85 hdd 9.09560 9.09560 osd.85 > 89 hdd 9.09560 9.09560 osd.89 > 90 hdd 10.91409 10.91409 osd.90 > 91 hdd 10.91409 10.91409 osd.91 > 96 hdd 10.91409 10.91409 osd.96 > 97 hdd 10.91409 10.91409 osd.97 > -5 316.52426 316.52426 host node2 > 2 hdd 9.09560 9.09560 osd.2 > 3 hdd 9.09560 9.09560 osd.3 > 7 hdd 9.09560 9.09560 osd.7 > 9 hdd 9.09560 9.09560 osd.9 > 14 hdd 9.09560 9.09560 osd.14 > 17 hdd 9.09560 9.09560 osd.17 > 18 hdd 9.09560 9.09560 osd.18 > 22 hdd 9.09560 9.09560 osd.22 > 24 hdd 9.09560 9.09560 osd.24 > 27 hdd 9.09560 9.09560 osd.27 > 30 hdd 9.09560 9.09560 osd.30 > 33 hdd 9.09560 9.09560 osd.33 > 36 hdd 9.09560 9.09560 osd.36 > 39 hdd 9.09560 9.09560 osd.39 > 42 hdd 9.09560 9.09560 osd.42 > 45 hdd 9.09560 9.09560 osd.45 > 48 hdd 9.09560 9.09560 osd.48 > 51 hdd 9.09560 9.09560 osd.51 > 54 hdd 9.09560 9.09560 osd.54 > 57 hdd 9.09560 9.09560 osd.57 > 60 hdd 9.09560 9.09560 osd.60 > 63 hdd 9.09560 9.09560 osd.63 > 66 hdd 9.09560 9.09560 osd.66 > 69 hdd 9.09560 9.09560 osd.69 > 72 hdd 9.09560 9.09560 osd.72 > 75 hdd 9.09560 9.09560 osd.75 > 78 hdd 9.09560 9.09560 osd.78 > 82 hdd 9.09560 9.09560 osd.82 > 84 hdd 9.09560 9.09560 osd.84 > 87 hdd 9.09560 9.09560 osd.87 > 92 hdd 10.91409 10.91409 osd.92 > 93 hdd 10.91409 10.91409 osd.93 > 100 hdd 10.91409 10.91409 osd.100 > 101 hdd 10.91409 10.91409 osd.101 > -13 371.07864 371.07864 host node3 > 102 hdd 10.91408 10.91408 osd.102 > 103 hdd 10.91408 10.91408 osd.103 > 104 hdd 10.91408 10.91408 osd.104 > 105 hdd 10.91408 10.91408 osd.105 > 106 hdd 10.91408 10.91408 osd.106 > 107 hdd 10.91408 10.91408 osd.107 > 108 hdd 10.91408 10.91408 osd.108 > 109 hdd 10.91408 10.91408 osd.109 > 110 hdd 10.91408 10.91408 osd.110 > 111 hdd 10.91408 10.91408 osd.111 > 112 hdd 10.91408 10.91408 osd.112 > 113 hdd 10.91408 10.91408 osd.113 > 114 hdd 10.91408 10.91408 osd.114 > 115 hdd 10.91408 10.91408 osd.115 > 116 hdd 10.91408 10.91408 osd.116 > 117 hdd 10.91408 10.91408 osd.117 > 118 hdd 10.91408 10.91408 osd.118 > 119 hdd 10.91408 10.91408 osd.119 > 120 hdd 10.91408 10.91408 osd.120 > 121 hdd 10.91408 10.91408 osd.121 > 122 hdd 10.91408 10.91408 osd.122 > 123 hdd 10.91408 10.91408 osd.123 > 124 hdd 10.91408 10.91408 osd.124 > 125 hdd 10.91408 10.91408 osd.125 > 126 hdd 10.91408 10.91408 osd.126 > 127 hdd 10.91408 10.91408 osd.127 > 128 hdd 10.91408 10.91408 osd.128 > 129 hdd 10.91408 10.91408 osd.129 > 130 hdd 10.91408 10.91408 osd.130 > 131 hdd 10.91408 10.91408 osd.131 > 132 hdd 10.91408 10.91408 osd.132 > 133 hdd 10.91408 10.91408 osd.133 > 134 hdd 10.91408 10.91408 osd.134 > 135 hdd 10.91408 10.91408 osd.135 > -16 371.07916 371.07916 host node4 > 136 hdd 10.91409 10.91409 osd.136 > 137 hdd 10.91409 10.91409 osd.137 > 138 hdd 10.91409 10.91409 osd.138 > 139 hdd 10.91409 10.91409 osd.139 > 140 hdd 10.91409 10.91409 osd.140 > 141 hdd 10.91409 10.91409 osd.141 > 142 hdd 10.91409 10.91409 osd.142 > 143 hdd 10.91409 10.91409 osd.143 > 144 hdd 10.91409 10.91409 osd.144 > 145 hdd 10.91409 10.91409 osd.145 > 146 hdd 10.91409 10.91409 osd.146 > 147 hdd 10.91409 10.91409 osd.147 > 148 hdd 10.91409 10.91409 osd.148 > 149 hdd 10.91409 10.91409 osd.149 > 150 hdd 10.91409 10.91409 osd.150 > 151 hdd 10.91409 10.91409 osd.151 > 152 hdd 10.91409 10.91409 osd.152 > 153 hdd 10.91409 10.91409 osd.153 > 154 hdd 10.91409 10.91409 osd.154 > 155 hdd 10.91409 10.91409 osd.155 > 156 hdd 10.91409 10.91409 osd.156 > 157 hdd 10.91409 10.91409 osd.157 > 158 hdd 10.91409 10.91409 osd.158 > 159 hdd 10.91409 10.91409 osd.159 > 160 hdd 10.91409 10.91409 osd.160 > 161 hdd 10.91409 10.91409 osd.161 > 162 hdd 10.91409 10.91409 osd.162 > 163 hdd 10.91409 10.91409 osd.163 > 164 hdd 10.91409 10.91409 osd.164 > 165 hdd 10.91409 10.91409 osd.165 > 166 hdd 10.91409 10.91409 osd.166 > 167 hdd 10.91409 10.91409 osd.167 > 168 hdd 10.91409 10.91409 osd.168 > 169 hdd 10.91409 10.91409 osd.169 > -19 371.07916 371.07916 host node5 > 170 hdd 10.91409 10.91409 osd.170 > 171 hdd 10.91409 10.91409 osd.171 > 173 hdd 10.91409 10.91409 osd.173 > 174 hdd 10.91409 10.91409 osd.174 > 175 hdd 10.91409 10.91409 osd.175 > 176 hdd 10.91409 10.91409 osd.176 > 177 hdd 10.91409 10.91409 osd.177 > 178 hdd 10.91409 10.91409 osd.178 > 179 hdd 10.91409 10.91409 osd.179 > 180 hdd 10.91409 10.91409 osd.180 > 181 hdd 10.91409 10.91409 osd.181 > 182 hdd 10.91409 10.91409 osd.182 > 183 hdd 10.91409 10.91409 osd.183 > 184 hdd 10.91409 10.91409 osd.184 > 185 hdd 10.91409 10.91409 osd.185 > 186 hdd 10.91409 10.91409 osd.186 > 187 hdd 10.91409 10.91409 osd.187 > 188 hdd 10.91409 10.91409 osd.188 > 189 hdd 10.91409 10.91409 osd.189 > 190 hdd 10.91409 10.91409 osd.190 > 191 hdd 10.91409 10.91409 osd.191 > 192 hdd 10.91409 10.91409 osd.192 > 193 hdd 10.91409 10.91409 osd.193 > 194 hdd 10.91409 10.91409 osd.194 > 195 hdd 10.91409 10.91409 osd.195 > 196 hdd 10.91409 10.91409 osd.196 > 197 hdd 10.91409 10.91409 osd.197 > 198 hdd 10.91409 10.91409 osd.198 > 199 hdd 10.91409 10.91409 osd.199 > 200 hdd 10.91409 10.91409 osd.200 > 201 hdd 10.91409 10.91409 osd.201 > 202 hdd 10.91409 10.91409 osd.202 > 203 hdd 10.91409 10.91409 osd.203 > 237 hdd 10.91409 10.91409 osd.237 > -22 371.07916 371.07916 host node6 > 172 hdd 10.91409 10.91409 osd.172 > 204 hdd 10.91409 10.91409 osd.204 > 205 hdd 10.91409 10.91409 osd.205 > 206 hdd 10.91409 10.91409 osd.206 > 207 hdd 10.91409 10.91409 osd.207 > 208 hdd 10.91409 10.91409 osd.208 > 209 hdd 10.91409 10.91409 osd.209 > 210 hdd 10.91409 10.91409 osd.210 > 211 hdd 10.91409 10.91409 osd.211 > 212 hdd 10.91409 10.91409 osd.212 > 213 hdd 10.91409 10.91409 osd.213 > 214 hdd 10.91409 10.91409 osd.214 > 215 hdd 10.91409 10.91409 osd.215 > 216 hdd 10.91409 10.91409 osd.216 > 217 hdd 10.91409 10.91409 osd.217 > 218 hdd 10.91409 10.91409 osd.218 > 219 hdd 10.91409 10.91409 osd.219 > 220 hdd 10.91409 10.91409 osd.220 > 221 hdd 10.91409 10.91409 osd.221 > 222 hdd 10.91409 10.91409 osd.222 > 223 hdd 10.91409 10.91409 osd.223 > 224 hdd 10.91409 10.91409 osd.224 > 225 hdd 10.91409 10.91409 osd.225 > 226 hdd 10.91409 10.91409 osd.226 > 227 hdd 10.91409 10.91409 osd.227 > 228 hdd 10.91409 10.91409 osd.228 > 229 hdd 10.91409 10.91409 osd.229 > 230 hdd 10.91409 10.91409 osd.230 > 231 hdd 10.91409 10.91409 osd.231 > 232 hdd 10.91409 10.91409 osd.232 > 233 hdd 10.91409 10.91409 osd.233 > 234 hdd 10.91409 10.91409 osd.234 > 235 hdd 10.91409 10.91409 osd.235 > 236 hdd 10.91409 10.91409 osd.236 > -28 371.07916 371.07916 host node7 > 50 hdd 10.91409 10.91409 osd.50 > 53 hdd 10.91409 10.91409 osd.53 > 62 hdd 10.91409 10.91409 osd.62 > 68 hdd 10.91409 10.91409 osd.68 > 74 hdd 10.91409 10.91409 osd.74 > 244 hdd 10.91409 10.91409 osd.244 > 245 hdd 10.91409 10.91409 osd.245 > 246 hdd 10.91409 10.91409 osd.246 > 247 hdd 10.91409 10.91409 osd.247 > 248 hdd 10.91409 10.91409 osd.248 > 249 hdd 10.91409 10.91409 osd.249 > 250 hdd 10.91409 10.91409 osd.250 > 251 hdd 10.91409 10.91409 osd.251 > 252 hdd 10.91409 10.91409 osd.252 > 253 hdd 10.91409 10.91409 osd.253 > 254 hdd 10.91409 10.91409 osd.254 > 255 hdd 10.91409 10.91409 osd.255 > 256 hdd 10.91409 10.91409 osd.256 > 257 hdd 10.91409 10.91409 osd.257 > 258 hdd 10.91409 10.91409 osd.258 > 259 hdd 10.91409 10.91409 osd.259 > 260 hdd 10.91409 10.91409 osd.260 > 261 hdd 10.91409 10.91409 osd.261 > 262 hdd 10.91409 10.91409 osd.262 > 263 hdd 10.91409 10.91409 osd.263 > 264 hdd 10.91409 10.91409 osd.264 > 265 hdd 10.91409 10.91409 osd.265 > 266 hdd 10.91409 10.91409 osd.266 > 267 hdd 10.91409 10.91409 osd.267 > 268 hdd 10.91409 10.91409 osd.268 > 269 hdd 10.91409 10.91409 osd.269 > 270 hdd 10.91409 10.91409 osd.270 > 271 hdd 10.91409 10.91409 osd.271 > 272 hdd 10.91409 10.91409 osd.272 > -25 371.07916 371.07916 host node8 > 40 hdd 10.91409 10.91409 osd.40 > 273 hdd 10.91409 10.91409 osd.273 > 274 hdd 10.91409 10.91409 osd.274 > 275 hdd 10.91409 10.91409 osd.275 > 276 hdd 10.91409 10.91409 osd.276 > 277 hdd 10.91409 10.91409 osd.277 > 278 hdd 10.91409 10.91409 osd.278 > 279 hdd 10.91409 10.91409 osd.279 > 280 hdd 10.91409 10.91409 osd.280 > 281 hdd 10.91409 10.91409 osd.281 > 282 hdd 10.91409 10.91409 osd.282 > 283 hdd 10.91409 10.91409 osd.283 > 284 hdd 10.91409 10.91409 osd.284 > 285 hdd 10.91409 10.91409 osd.285 > 286 hdd 10.91409 10.91409 osd.286 > 287 hdd 10.91409 10.91409 osd.287 > 288 hdd 10.91409 10.91409 osd.288 > 289 hdd 10.91409 10.91409 osd.289 > 290 hdd 10.91409 10.91409 osd.290 > 291 hdd 10.91409 10.91409 osd.291 > 292 hdd 10.91409 10.91409 osd.292 > 293 hdd 10.91409 10.91409 osd.293 > 294 hdd 10.91409 10.91409 osd.294 > 295 hdd 10.91409 10.91409 osd.295 > 296 hdd 10.91409 10.91409 osd.296 > 297 hdd 10.91409 10.91409 osd.297 > 298 hdd 10.91409 10.91409 osd.298 > 299 hdd 10.91409 10.91409 osd.299 > 300 hdd 10.91409 10.91409 osd.300 > 301 hdd 10.91409 10.91409 osd.301 > 302 hdd 10.91409 10.91409 osd.302 > 303 hdd 10.91409 10.91409 osd.303 > 304 hdd 10.91409 10.91409 osd.304 > 305 hdd 10.91409 10.91409 osd.305 > > bye > wolfgang > > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
