thx Paul - I suspect these shadow trees causing this misbehaviour.
I have a second luminous cluster where these balancer settings work as expected 
- this working one has hdd+ssd osds

i cannot use the upmap balancer because of some jewel-krbd clients - at least 
they are being reported as jewel clients

    "client": {
        "group": {
            "features": "0x7010fb86aa42ada",
            "release": "jewel",
            "num": 1
        },
        "group": {
            "features": "0x27018fb86aa42ada",
            "release": "jewel",
            "num": 3
        },

is there a good way to decode the "features" value?

wolfgang


Am 26.06.2019 um 13:29 schrieb Paul Emmerich:
Device classes are implemented with magic invisible crush trees; you've got two completely independent trees internally: one for crush rules mapping to HDDs, one to legacy crush rules not specifying a device class.

The balancer *should* be aware of this and ignore it, but I'm not sure about the state of the balancer on Luminous. There were quite a few problems in older versions, lots of them have been fixed in backports.

The upmap balancer is much better than the crush-compat balancer, but it requires all clients to run Luminous or later.


Paul

--
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io <http://www.croit.io>
Tel: +49 89 1896585 90


On Wed, Jun 26, 2019 at 10:21 AM Wolfgang Lendl <wolfgang.le...@meduniwien.ac.at <mailto:wolfgang.le...@meduniwien.ac.at>> wrote:

    Hi,

    tried to enable the ceph balancer on a 12.2.12 cluster and got this:

    mgr[balancer] Some osds belong to multiple subtrees: [0, 1, 2, 3, 4, 5, 6, 
7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 
47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 
67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 
87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 
105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 
121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 
137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 
153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 
169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 
185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 
201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 
217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 
233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 
249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 
265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 
281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 
297, 298, 299, 300, 301, 302, 303, 304, 305]

    I'm not aware of any additional subtree - maybe someone can enlighten me:

    ceph balancer status
    {
         "active": true,
         "plans": [],
         "mode": "crush-compat"
    }

    ceph osd crush tree
    ID  CLASS WEIGHT     (compat)  TYPE NAME
      -1       3176.04785           root default
      -7        316.52490 316.52490     host node0
       0   hdd    9.09560   9.09560         osd.0
       4   hdd    9.09560   9.09560         osd.4
       8   hdd    9.09560   9.09560         osd.8
      10   hdd    9.09560   9.09560         osd.10
      12   hdd    9.09560   9.09560         osd.12
      16   hdd    9.09560   9.09560         osd.16
      20   hdd    9.09560   9.09560         osd.20
      21   hdd    9.09560   9.09560         osd.21
      26   hdd    9.09560   9.09560         osd.26
      29   hdd    9.09560   9.09560         osd.29
      31   hdd    9.09560   9.09560         osd.31
      35   hdd    9.09560   9.09560         osd.35
      37   hdd    9.09560   9.09560         osd.37
      44   hdd    9.09560   9.09560         osd.44
      47   hdd    9.09560   9.09560         osd.47
      56   hdd    9.09560   9.09560         osd.56
      59   hdd    9.09560   9.09560         osd.59
      65   hdd    9.09560   9.09560         osd.65
      71   hdd    9.09560   9.09560         osd.71
      77   hdd    9.09560   9.09560         osd.77
      80   hdd    9.09560   9.09560         osd.80
      83   hdd    9.09569   9.09569         osd.83
      86   hdd    9.09560   9.09560         osd.86
      88   hdd    9.09560   9.09560         osd.88
      94   hdd   10.91409  10.91409         osd.94
      95   hdd   10.91409  10.91409         osd.95
      98   hdd   10.91409  10.91409         osd.98
      99   hdd   10.91409  10.91409         osd.99
    238   hdd    9.09569   9.09569         osd.238
    239   hdd    9.09569   9.09569         osd.239
    240   hdd    9.09569   9.09569         osd.240
    241   hdd    9.09569   9.09569         osd.241
    242   hdd    9.09569   9.09569         osd.242
    243   hdd    9.09569   9.09569         osd.243
      -3        316.52426 316.52426     host node1
       1   hdd    9.09560   9.09560         osd.1
       5   hdd    9.09560   9.09560         osd.5
       6   hdd    9.09560   9.09560         osd.6
      11   hdd    9.09560   9.09560         osd.11
      13   hdd    9.09560   9.09560         osd.13
      15   hdd    9.09560   9.09560         osd.15
      19   hdd    9.09560   9.09560         osd.19
      23   hdd    9.09560   9.09560         osd.23
      25   hdd    9.09560   9.09560         osd.25
      28   hdd    9.09560   9.09560         osd.28
      32   hdd    9.09560   9.09560         osd.32
      34   hdd    9.09560   9.09560         osd.34
      38   hdd    9.09560   9.09560         osd.38
      41   hdd    9.09560   9.09560         osd.41
      43   hdd    9.09560   9.09560         osd.43
      46   hdd    9.09560   9.09560         osd.46
      49   hdd    9.09560   9.09560         osd.49
      52   hdd    9.09560   9.09560         osd.52
      55   hdd    9.09560   9.09560         osd.55
      58   hdd    9.09560   9.09560         osd.58
      61   hdd    9.09560   9.09560         osd.61
      64   hdd    9.09560   9.09560         osd.64
      67   hdd    9.09560   9.09560         osd.67
      70   hdd    9.09560   9.09560         osd.70
      73   hdd    9.09560   9.09560         osd.73
      76   hdd    9.09560   9.09560         osd.76
      79   hdd    9.09560   9.09560         osd.79
      81   hdd    9.09560   9.09560         osd.81
      85   hdd    9.09560   9.09560         osd.85
      89   hdd    9.09560   9.09560         osd.89
      90   hdd   10.91409  10.91409         osd.90
      91   hdd   10.91409  10.91409         osd.91
      96   hdd   10.91409  10.91409         osd.96
      97   hdd   10.91409  10.91409         osd.97
      -5        316.52426 316.52426     host node2
       2   hdd    9.09560   9.09560         osd.2
       3   hdd    9.09560   9.09560         osd.3
       7   hdd    9.09560   9.09560         osd.7
       9   hdd    9.09560   9.09560         osd.9
      14   hdd    9.09560   9.09560         osd.14
      17   hdd    9.09560   9.09560         osd.17
      18   hdd    9.09560   9.09560         osd.18
      22   hdd    9.09560   9.09560         osd.22
      24   hdd    9.09560   9.09560         osd.24
      27   hdd    9.09560   9.09560         osd.27
      30   hdd    9.09560   9.09560         osd.30
      33   hdd    9.09560   9.09560         osd.33
      36   hdd    9.09560   9.09560         osd.36
      39   hdd    9.09560   9.09560         osd.39
      42   hdd    9.09560   9.09560         osd.42
      45   hdd    9.09560   9.09560         osd.45
      48   hdd    9.09560   9.09560         osd.48
      51   hdd    9.09560   9.09560         osd.51
      54   hdd    9.09560   9.09560         osd.54
      57   hdd    9.09560   9.09560         osd.57
      60   hdd    9.09560   9.09560         osd.60
      63   hdd    9.09560   9.09560         osd.63
      66   hdd    9.09560   9.09560         osd.66
      69   hdd    9.09560   9.09560         osd.69
      72   hdd    9.09560   9.09560         osd.72
      75   hdd    9.09560   9.09560         osd.75
      78   hdd    9.09560   9.09560         osd.78
      82   hdd    9.09560   9.09560         osd.82
      84   hdd    9.09560   9.09560         osd.84
      87   hdd    9.09560   9.09560         osd.87
      92   hdd   10.91409  10.91409         osd.92
      93   hdd   10.91409  10.91409         osd.93
    100   hdd   10.91409  10.91409         osd.100
    101   hdd   10.91409  10.91409         osd.101
    -13        371.07864 371.07864     host node3
    102   hdd   10.91408  10.91408         osd.102
    103   hdd   10.91408  10.91408         osd.103
    104   hdd   10.91408  10.91408         osd.104
    105   hdd   10.91408  10.91408         osd.105
    106   hdd   10.91408  10.91408         osd.106
    107   hdd   10.91408  10.91408         osd.107
    108   hdd   10.91408  10.91408         osd.108
    109   hdd   10.91408  10.91408         osd.109
    110   hdd   10.91408  10.91408         osd.110
    111   hdd   10.91408  10.91408         osd.111
    112   hdd   10.91408  10.91408         osd.112
    113   hdd   10.91408  10.91408         osd.113
    114   hdd   10.91408  10.91408         osd.114
    115   hdd   10.91408  10.91408         osd.115
    116   hdd   10.91408  10.91408         osd.116
    117   hdd   10.91408  10.91408         osd.117
    118   hdd   10.91408  10.91408         osd.118
    119   hdd   10.91408  10.91408         osd.119
    120   hdd   10.91408  10.91408         osd.120
    121   hdd   10.91408  10.91408         osd.121
    122   hdd   10.91408  10.91408         osd.122
    123   hdd   10.91408  10.91408         osd.123
    124   hdd   10.91408  10.91408         osd.124
    125   hdd   10.91408  10.91408         osd.125
    126   hdd   10.91408  10.91408         osd.126
    127   hdd   10.91408  10.91408         osd.127
    128   hdd   10.91408  10.91408         osd.128
    129   hdd   10.91408  10.91408         osd.129
    130   hdd   10.91408  10.91408         osd.130
    131   hdd   10.91408  10.91408         osd.131
    132   hdd   10.91408  10.91408         osd.132
    133   hdd   10.91408  10.91408         osd.133
    134   hdd   10.91408  10.91408         osd.134
    135   hdd   10.91408  10.91408         osd.135
    -16        371.07916 371.07916     host node4
    136   hdd   10.91409  10.91409         osd.136
    137   hdd   10.91409  10.91409         osd.137
    138   hdd   10.91409  10.91409         osd.138
    139   hdd   10.91409  10.91409         osd.139
    140   hdd   10.91409  10.91409         osd.140
    141   hdd   10.91409  10.91409         osd.141
    142   hdd   10.91409  10.91409         osd.142
    143   hdd   10.91409  10.91409         osd.143
    144   hdd   10.91409  10.91409         osd.144
    145   hdd   10.91409  10.91409         osd.145
    146   hdd   10.91409  10.91409         osd.146
    147   hdd   10.91409  10.91409         osd.147
    148   hdd   10.91409  10.91409         osd.148
    149   hdd   10.91409  10.91409         osd.149
    150   hdd   10.91409  10.91409         osd.150
    151   hdd   10.91409  10.91409         osd.151
    152   hdd   10.91409  10.91409         osd.152
    153   hdd   10.91409  10.91409         osd.153
    154   hdd   10.91409  10.91409         osd.154
    155   hdd   10.91409  10.91409         osd.155
    156   hdd   10.91409  10.91409         osd.156
    157   hdd   10.91409  10.91409         osd.157
    158   hdd   10.91409  10.91409         osd.158
    159   hdd   10.91409  10.91409         osd.159
    160   hdd   10.91409  10.91409         osd.160
    161   hdd   10.91409  10.91409         osd.161
    162   hdd   10.91409  10.91409         osd.162
    163   hdd   10.91409  10.91409         osd.163
    164   hdd   10.91409  10.91409         osd.164
    165   hdd   10.91409  10.91409         osd.165
    166   hdd   10.91409  10.91409         osd.166
    167   hdd   10.91409  10.91409         osd.167
    168   hdd   10.91409  10.91409         osd.168
    169   hdd   10.91409  10.91409         osd.169
    -19        371.07916 371.07916     host node5
    170   hdd   10.91409  10.91409         osd.170
    171   hdd   10.91409  10.91409         osd.171
    173   hdd   10.91409  10.91409         osd.173
    174   hdd   10.91409  10.91409         osd.174
    175   hdd   10.91409  10.91409         osd.175
    176   hdd   10.91409  10.91409         osd.176
    177   hdd   10.91409  10.91409         osd.177
    178   hdd   10.91409  10.91409         osd.178
    179   hdd   10.91409  10.91409         osd.179
    180   hdd   10.91409  10.91409         osd.180
    181   hdd   10.91409  10.91409         osd.181
    182   hdd   10.91409  10.91409         osd.182
    183   hdd   10.91409  10.91409         osd.183
    184   hdd   10.91409  10.91409         osd.184
    185   hdd   10.91409  10.91409         osd.185
    186   hdd   10.91409  10.91409         osd.186
    187   hdd   10.91409  10.91409         osd.187
    188   hdd   10.91409  10.91409         osd.188
    189   hdd   10.91409  10.91409         osd.189
    190   hdd   10.91409  10.91409         osd.190
    191   hdd   10.91409  10.91409         osd.191
    192   hdd   10.91409  10.91409         osd.192
    193   hdd   10.91409  10.91409         osd.193
    194   hdd   10.91409  10.91409         osd.194
    195   hdd   10.91409  10.91409         osd.195
    196   hdd   10.91409  10.91409         osd.196
    197   hdd   10.91409  10.91409         osd.197
    198   hdd   10.91409  10.91409         osd.198
    199   hdd   10.91409  10.91409         osd.199
    200   hdd   10.91409  10.91409         osd.200
    201   hdd   10.91409  10.91409         osd.201
    202   hdd   10.91409  10.91409         osd.202
    203   hdd   10.91409  10.91409         osd.203
    237   hdd   10.91409  10.91409         osd.237
    -22        371.07916 371.07916     host node6
    172   hdd   10.91409  10.91409         osd.172
    204   hdd   10.91409  10.91409         osd.204
    205   hdd   10.91409  10.91409         osd.205
    206   hdd   10.91409  10.91409         osd.206
    207   hdd   10.91409  10.91409         osd.207
    208   hdd   10.91409  10.91409         osd.208
    209   hdd   10.91409  10.91409         osd.209
    210   hdd   10.91409  10.91409         osd.210
    211   hdd   10.91409  10.91409         osd.211
    212   hdd   10.91409  10.91409         osd.212
    213   hdd   10.91409  10.91409         osd.213
    214   hdd   10.91409  10.91409         osd.214
    215   hdd   10.91409  10.91409         osd.215
    216   hdd   10.91409  10.91409         osd.216
    217   hdd   10.91409  10.91409         osd.217
    218   hdd   10.91409  10.91409         osd.218
    219   hdd   10.91409  10.91409         osd.219
    220   hdd   10.91409  10.91409         osd.220
    221   hdd   10.91409  10.91409         osd.221
    222   hdd   10.91409  10.91409         osd.222
    223   hdd   10.91409  10.91409         osd.223
    224   hdd   10.91409  10.91409         osd.224
    225   hdd   10.91409  10.91409         osd.225
    226   hdd   10.91409  10.91409         osd.226
    227   hdd   10.91409  10.91409         osd.227
    228   hdd   10.91409  10.91409         osd.228
    229   hdd   10.91409  10.91409         osd.229
    230   hdd   10.91409  10.91409         osd.230
    231   hdd   10.91409  10.91409         osd.231
    232   hdd   10.91409  10.91409         osd.232
    233   hdd   10.91409  10.91409         osd.233
    234   hdd   10.91409  10.91409         osd.234
    235   hdd   10.91409  10.91409         osd.235
    236   hdd   10.91409  10.91409         osd.236
    -28        371.07916 371.07916     host node7
      50   hdd   10.91409  10.91409         osd.50
      53   hdd   10.91409  10.91409         osd.53
      62   hdd   10.91409  10.91409         osd.62
      68   hdd   10.91409  10.91409         osd.68
      74   hdd   10.91409  10.91409         osd.74
    244   hdd   10.91409  10.91409         osd.244
    245   hdd   10.91409  10.91409         osd.245
    246   hdd   10.91409  10.91409         osd.246
    247   hdd   10.91409  10.91409         osd.247
    248   hdd   10.91409  10.91409         osd.248
    249   hdd   10.91409  10.91409         osd.249
    250   hdd   10.91409  10.91409         osd.250
    251   hdd   10.91409  10.91409         osd.251
    252   hdd   10.91409  10.91409         osd.252
    253   hdd   10.91409  10.91409         osd.253
    254   hdd   10.91409  10.91409         osd.254
    255   hdd   10.91409  10.91409         osd.255
    256   hdd   10.91409  10.91409         osd.256
    257   hdd   10.91409  10.91409         osd.257
    258   hdd   10.91409  10.91409         osd.258
    259   hdd   10.91409  10.91409         osd.259
    260   hdd   10.91409  10.91409         osd.260
    261   hdd   10.91409  10.91409         osd.261
    262   hdd   10.91409  10.91409         osd.262
    263   hdd   10.91409  10.91409         osd.263
    264   hdd   10.91409  10.91409         osd.264
    265   hdd   10.91409  10.91409         osd.265
    266   hdd   10.91409  10.91409         osd.266
    267   hdd   10.91409  10.91409         osd.267
    268   hdd   10.91409  10.91409         osd.268
    269   hdd   10.91409  10.91409         osd.269
    270   hdd   10.91409  10.91409         osd.270
    271   hdd   10.91409  10.91409         osd.271
    272   hdd   10.91409  10.91409         osd.272
    -25        371.07916 371.07916     host node8
      40   hdd   10.91409  10.91409         osd.40
    273   hdd   10.91409  10.91409         osd.273
    274   hdd   10.91409  10.91409         osd.274
    275   hdd   10.91409  10.91409         osd.275
    276   hdd   10.91409  10.91409         osd.276
    277   hdd   10.91409  10.91409         osd.277
    278   hdd   10.91409  10.91409         osd.278
    279   hdd   10.91409  10.91409         osd.279
    280   hdd   10.91409  10.91409         osd.280
    281   hdd   10.91409  10.91409         osd.281
    282   hdd   10.91409  10.91409         osd.282
    283   hdd   10.91409  10.91409         osd.283
    284   hdd   10.91409  10.91409         osd.284
    285   hdd   10.91409  10.91409         osd.285
    286   hdd   10.91409  10.91409         osd.286
    287   hdd   10.91409  10.91409         osd.287
    288   hdd   10.91409  10.91409         osd.288
    289   hdd   10.91409  10.91409         osd.289
    290   hdd   10.91409  10.91409         osd.290
    291   hdd   10.91409  10.91409         osd.291
    292   hdd   10.91409  10.91409         osd.292
    293   hdd   10.91409  10.91409         osd.293
    294   hdd   10.91409  10.91409         osd.294
    295   hdd   10.91409  10.91409         osd.295
    296   hdd   10.91409  10.91409         osd.296
    297   hdd   10.91409  10.91409         osd.297
    298   hdd   10.91409  10.91409         osd.298
    299   hdd   10.91409  10.91409         osd.299
    300   hdd   10.91409  10.91409         osd.300
    301   hdd   10.91409  10.91409         osd.301
    302   hdd   10.91409  10.91409         osd.302
    303   hdd   10.91409  10.91409         osd.303
    304   hdd   10.91409  10.91409         osd.304
    305   hdd   10.91409  10.91409         osd.305

    bye
    wolfgang

    _______________________________________________
    ceph-users mailing list
    ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
    http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to