On 2024/11/16 18:38, Anthony D'Atri wrote:
Disabling mclock as described here https://docs.ceph.com/en/reef/rados/configuration/mclock-config-ref/ might help

I cannot see any option that allows me to disable mclock...


Also, you have a small cluster with a bunch of small OSDs.  Please send `ceph health detail`

# ceph health detail
HEALTH_WARN 1 pool(s) have no replicas configured
[WRN] POOL_NO_REDUNDANCY: 1 pool(s) have no replicas configured
    pool 'backups' has no replicas configured

The backups pool is on spinners and 1 copy is sufficient since we're also replicating this offsite in real time.  Otherwise too many copies take too much space.


Please also send `ceph osd df` and `ceph osd crush dump`
NodeC:~# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE   DATA     OMAP META      AVAIL    %USE   VAR   PGS  STATUS  2    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.1 TiB   16 MiB   2.7 GiB  746 GiB  60.97  1.23   68      up  3    hdd  1.86029   1.00000  1.9 TiB   949 GiB  902 GiB   12 MiB   2.6 GiB  961 GiB  49.68  1.00   57      up  4    hdd  1.86029   1.00000  1.9 TiB   848 GiB  800 GiB   10 MiB   2.3 GiB  1.0 TiB  44.37  0.89   51      up  5    hdd  1.86589   1.00000  1.9 TiB   1.0 TiB  982 GiB   15 MiB   2.4 GiB  881 GiB  53.87  1.08   66      up  0    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0    0    down  1    ssd  0.28319   1.00000  290 GiB   123 GiB  122 GiB  2.3 MiB   1.2 GiB  167 GiB  42.38  0.85   46      up 36    ssd        0   1.00000  290 GiB   1.3 GiB   23 MiB  2.8 MiB   1.2 GiB  289 GiB   0.44  0.01   13      up 37    ssd        0   1.00000  290 GiB   1.1 GiB   23 MiB  2.5 MiB   1.1 GiB  289 GiB   0.38  0.01   18      up 38    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   41      up 39    ssd        0         0      0 B       0 B      0 B      0 B       0 B      0 B      0     0   86      up 40    ssd  3.30690   1.00000  3.3 TiB   1.7 TiB  1.7 TiB   23 MiB   4.8 GiB  1.6 TiB  50.49  1.02  424      up 10    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  990 GiB   13 MiB   2.8 GiB  873 GiB  54.19  1.09   64      up 11    hdd  1.86029   1.00000  1.9 TiB   1.3 TiB  1.2 TiB   15 MiB   3.2 GiB  625 GiB  67.20  1.35   72      up 26    hdd  1.86029   1.00000  1.9 TiB   843 GiB  801 GiB   12 MiB   2.3 GiB  1.0 TiB  44.25  0.89   50      up 27    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   16 MiB   2.7 GiB  793 GiB  58.38  1.18   70      up  6    ssd  0.28319   1.00000  290 GiB   115 GiB  115 GiB  2.3 MiB   743 MiB  175 GiB  39.82  0.80   28      up  7    ssd  0.28319   1.00000  290 GiB    94 GiB   93 GiB  2.0 MiB   921 MiB  196 GiB  32.51  0.65   23      up  8    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   963 MiB  187 GiB  35.46  0.71   27      up  9    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.1 GiB  159 GiB  45.28  0.91   33      up 24    ssd  0.28319   1.00000  290 GiB   102 GiB  101 GiB  2.1 MiB   719 MiB  188 GiB  35.10  0.71   26      up 25    ssd  0.28319   1.00000  290 GiB   152 GiB  151 GiB  2.8 MiB 1020 MiB  138 GiB  52.52  1.06   39      up 41    ssd  3.49309   1.00000  3.5 TiB   1.6 TiB  1.6 TiB   22 MiB   5.4 GiB  1.9 TiB  44.58  0.90  411      up 14    hdd  1.59999   1.00000  1.9 TiB   1.0 TiB  1.0 TiB   13 MiB   2.4 GiB  837 GiB  56.04  1.13   63      up 15    hdd  1.86029   1.00000  1.9 TiB  1017 GiB  975 GiB   13 MiB   2.1 GiB  888 GiB  53.36  1.07   60      up 16    hdd  1.86029   1.00000  1.9 TiB   990 GiB  948 GiB   13 MiB   2.2 GiB  915 GiB  51.97  1.05   59      up 17    hdd  1.86029   1.00000  1.9 TiB   1.0 TiB  992 GiB   14 MiB   2.5 GiB  871 GiB  54.28  1.09   64      up 12    ssd  0.28319   1.00000  290 GiB   122 GiB  121 GiB  2.3 MiB   1.1 GiB  168 GiB  41.96  0.84   31      up 13    ssd  0.28319   1.00000  290 GiB   145 GiB  144 GiB  2.6 MiB   1.2 GiB  145 GiB  49.95  1.01   37      up 28    ssd  0.28319   1.00000  290 GiB   142 GiB  141 GiB  2.6 MiB   1.1 GiB  148 GiB  48.89  0.98   35      up 29    ssd  0.28319   1.00000  290 GiB   109 GiB  108 GiB  2.2 MiB   1.3 GiB  181 GiB  37.58  0.76   29      up 30    ssd  0.28319   1.00000  290 GiB   146 GiB  145 GiB  2.7 MiB   1.3 GiB  144 GiB  50.37  1.01   37      up 31    ssd  0.28319   1.00000  290 GiB   131 GiB  130 GiB  2.5 MiB   1.3 GiB  159 GiB  45.24  0.91   34      up 43    ssd  3.30690   1.00000  3.3 TiB   1.4 TiB  1.4 TiB   19 MiB   4.0 GiB  1.9 TiB  42.50  0.86  353      up 20    hdd  1.86029   1.00000  1.9 TiB   1.2 TiB  1.2 TiB   12 MiB   2.8 GiB  659 GiB  65.38  1.32   67      up 21    hdd  1.86029   1.00000  1.9 TiB   1.1 TiB  1.0 TiB   15 MiB   2.8 GiB  815 GiB  57.20  1.15   68      up 22    hdd  1.86029   1.00000  1.9 TiB   878 GiB  836 GiB   13 MiB   2.4 GiB  1.0 TiB  46.10  0.93   54      up 23    hdd  1.86029   1.00000  1.9 TiB  1018 GiB  977 GiB   14 MiB   2.8 GiB  886 GiB  53.46  1.08   59      up 18    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   1.3 GiB  175 GiB  39.74  0.80   29      up 19    ssd  0.28319   1.00000  290 GiB   115 GiB  114 GiB  2.3 MiB   961 MiB  175 GiB  39.59  0.80   28      up 32    ssd  0.28319   1.00000  290 GiB   103 GiB  102 GiB  2.1 MiB   1.0 GiB  187 GiB  35.65  0.72   26      up 33    ssd  0.28319   1.00000  290 GiB    98 GiB   97 GiB  2.1 MiB   1.4 GiB  192 GiB  33.92  0.68   26      up 34    ssd  0.28319   1.00000  290 GiB   141 GiB  140 GiB  2.6 MiB   987 MiB  149 GiB  48.49  0.98   35      up 35    ssd  0.28319   1.00000  290 GiB   116 GiB  114 GiB  2.3 MiB   1.3 GiB  174 GiB  39.92  0.80   31      up 42    ssd  3.49309   1.00000  3.5 TiB   1.4 TiB  1.4 TiB   20 MiB   4.5 GiB  2.1 TiB  40.45  0.81  372      up                        TOTAL   49 TiB    25 TiB   24 TiB  351 MiB    83 GiB   25 TiB  49.67

MIN/MAX VAR: 0.01/1.35  STDDEV: 13.87

NodeC:~# ceph osd crush dump
{
    "devices": [
        {
            "id": 0,
            "name": "osd.0",
            "class": "ssd"
        },
        {
            "id": 1,
            "name": "osd.1",
            "class": "ssd"
        },
        {
            "id": 2,
            "name": "osd.2",
            "class": "hdd"
        },
        {
            "id": 3,
            "name": "osd.3",
            "class": "hdd"
        },
        {
            "id": 4,
            "name": "osd.4",
            "class": "hdd"
        },
        {
            "id": 5,
            "name": "osd.5",
            "class": "hdd"
        },
        {
            "id": 6,
            "name": "osd.6",
            "class": "ssd"
        },
        {
            "id": 7,
            "name": "osd.7",
            "class": "ssd"
        },
        {
            "id": 8,
            "name": "osd.8",
            "class": "ssd"
        },
        {
            "id": 9,
            "name": "osd.9",
            "class": "ssd"
        },
        {
            "id": 10,
            "name": "osd.10",
            "class": "hdd"
        },
        {
            "id": 11,
            "name": "osd.11",
            "class": "hdd"
        },
        {
            "id": 12,
            "name": "osd.12",
            "class": "ssd"
        },
        {
            "id": 13,
            "name": "osd.13",
            "class": "ssd"
        },
        {
            "id": 14,
            "name": "osd.14",
            "class": "hdd"
        },
        {
            "id": 15,
            "name": "osd.15",
            "class": "hdd"
        },
        {
            "id": 16,
            "name": "osd.16",
            "class": "hdd"
        },
        {
            "id": 17,
            "name": "osd.17",
            "class": "hdd"
        },
        {
            "id": 18,
            "name": "osd.18",
            "class": "ssd"
        },
        {
            "id": 19,
            "name": "osd.19",
            "class": "ssd"
        },
        {
            "id": 20,
            "name": "osd.20",
            "class": "hdd"
        },
        {
            "id": 21,
            "name": "osd.21",
            "class": "hdd"
        },
        {
            "id": 22,
            "name": "osd.22",
            "class": "hdd"
        },
        {
            "id": 23,
            "name": "osd.23",
            "class": "hdd"
        },
        {
            "id": 24,
            "name": "osd.24",
            "class": "ssd"
        },
        {
            "id": 25,
            "name": "osd.25",
            "class": "ssd"
        },
        {
            "id": 26,
            "name": "osd.26",
            "class": "hdd"
        },
        {
            "id": 27,
            "name": "osd.27",
            "class": "hdd"
        },
        {
            "id": 28,
            "name": "osd.28",
            "class": "ssd"
        },
        {
            "id": 29,
            "name": "osd.29",
            "class": "ssd"
        },
        {
            "id": 30,
            "name": "osd.30",
            "class": "ssd"
        },
        {
            "id": 31,
            "name": "osd.31",
            "class": "ssd"
        },
        {
            "id": 32,
            "name": "osd.32",
            "class": "ssd"
        },
        {
            "id": 33,
            "name": "osd.33",
            "class": "ssd"
        },
        {
            "id": 34,
            "name": "osd.34",
            "class": "ssd"
        },
        {
            "id": 35,
            "name": "osd.35",
            "class": "ssd"
        },
        {
            "id": 36,
            "name": "osd.36",
            "class": "ssd"
        },
        {
            "id": 37,
            "name": "osd.37",
            "class": "ssd"
        },
        {
            "id": 38,
            "name": "osd.38",
            "class": "ssd"
        },
        {
            "id": 39,
            "name": "osd.39",
            "class": "ssd"
        },
        {
            "id": 40,
            "name": "osd.40",
            "class": "ssd"
        },
        {
            "id": 41,
            "name": "osd.41",
            "class": "ssd"
        },
        {
            "id": 42,
            "name": "osd.42",
            "class": "ssd"
        },
        {
            "id": 43,
            "name": "osd.43",
            "class": "ssd"
        }
    ],
    "types": [
        {
            "type_id": 0,
            "name": "osd"
        },
        {
            "type_id": 1,
            "name": "host"
        },
        {
            "type_id": 2,
            "name": "chassis"
        },
        {
            "type_id": 3,
            "name": "rack"
        },
        {
            "type_id": 4,
            "name": "row"
        },
        {
            "type_id": 5,
            "name": "pdu"
        },
        {
            "type_id": 6,
            "name": "pod"
        },
        {
            "type_id": 7,
            "name": "room"
        },
        {
            "type_id": 8,
            "name": "datacenter"
        },
        {
            "type_id": 9,
            "name": "zone"
        },
        {
            "type_id": 10,
            "name": "region"
        },
        {
            "type_id": 11,
            "name": "root"
        }
    ],
    "buckets": [
        {
            "id": -1,
            "name": "default",
            "type_id": 11,
            "type_name": "root",
            "weight": 3177873,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -3,
                    "weight": 723311,
                    "pos": 0
                },
                {
                    "id": -7,
                    "weight": 827941,
                    "pos": 1
                },
                {
                    "id": -10,
                    "weight": 798680,
                    "pos": 2
                },
                {
                    "id": -13,
                    "weight": 827941,
                    "pos": 3
                }
            ]
        },
        {
            "id": -2,
            "name": "default~ssd",
            "type_id": 11,
            "type_name": "root",
            "weight": 1243909,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -4,
                    "weight": 235280,
                    "pos": 0
                },
                {
                    "id": -8,
                    "weight": 340277,
                    "pos": 1
                },
                {
                    "id": -11,
                    "weight": 328075,
                    "pos": 2
                },
                {
                    "id": -14,
                    "weight": 340277,
                    "pos": 3
                }
            ]
        },
        {
            "id": -3,
            "name": "FT1-NodeA",
            "type_id": 1,
            "type_name": "host",
            "weight": 723311,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 1,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 36,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": 37,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": 0,
                    "weight": 0,
                    "pos": 3
                },
                {
                    "id": 38,
                    "weight": 0,
                    "pos": 4
                },
                {
                    "id": 39,
                    "weight": 0,
                    "pos": 5
                },
                {
                    "id": 2,
                    "weight": 121916,
                    "pos": 6
                },
                {
                    "id": 3,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 4,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 40,
                    "weight": 216721,
                    "pos": 9
                },
                {
                    "id": 5,
                    "weight": 122283,
                    "pos": 10
                }
            ]
        },
        {
            "id": -4,
            "name": "FT1-NodeA~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 235280,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 1,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 36,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": 37,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": 0,
                    "weight": 0,
                    "pos": 3
                },
                {
                    "id": 38,
                    "weight": 0,
                    "pos": 4
                },
                {
                    "id": 39,
                    "weight": 0,
                    "pos": 5
                },
                {
                    "id": 40,
                    "weight": 216721,
                    "pos": 6
                }
            ]
        },
        {
            "id": -5,
            "name": "FT1-NodeA~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 488031,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 2,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 3,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 4,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 5,
                    "weight": 122283,
                    "pos": 3
                }
            ]
        },
        {
            "id": -6,
            "name": "default~hdd",
            "type_id": 11,
            "type_name": "root",
            "weight": 1933964,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -5,
                    "weight": 488031,
                    "pos": 0
                },
                {
                    "id": -9,
                    "weight": 487664,
                    "pos": 1
                },
                {
                    "id": -12,
                    "weight": 470605,
                    "pos": 2
                },
                {
                    "id": -15,
                    "weight": 487664,
                    "pos": 3
                }
            ]
        },
        {
            "id": -7,
            "name": "FT1-NodeB",
            "type_id": 1,
            "type_name": "host",
            "weight": 827941,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 6,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 24,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 25,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 11,
                    "weight": 121916,
                    "pos": 3
                },
                {
                    "id": 10,
                    "weight": 121916,
                    "pos": 4
                },
                {
                    "id": 7,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 8,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 9,
                    "weight": 18559,
                    "pos": 7
                },
                {
                    "id": 26,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 27,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 41,
                    "weight": 228923,
                    "pos": 10
                }
            ]
        },
        {
            "id": -8,
            "name": "FT1-NodeB~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 340277,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 6,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 24,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 25,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 7,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 8,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 9,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 41,
                    "weight": 228923,
                    "pos": 6
                }
            ]
        },
        {
            "id": -9,
            "name": "FT1-NodeB~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 487664,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 11,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 10,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 26,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 27,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -10,
            "name": "FT1-NodeC",
            "type_id": 1,
            "type_name": "host",
            "weight": 798680,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 13,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 28,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 29,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 14,
                    "weight": 104857,
                    "pos": 3
                },
                {
                    "id": 12,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 30,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 31,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 15,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 16,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 17,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 43,
                    "weight": 216721,
                    "pos": 10
                }
            ]
        },
        {
            "id": -11,
            "name": "FT1-NodeC~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 328075,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 13,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 28,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 29,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 12,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 30,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 31,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 43,
                    "weight": 216721,
                    "pos": 6
                }
            ]
        },
        {
            "id": -12,
            "name": "FT1-NodeC~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 470605,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 14,
                    "weight": 104857,
                    "pos": 0
                },
                {
                    "id": 15,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 16,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 17,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -13,
            "name": "FT1-NodeD",
            "type_id": 1,
            "type_name": "host",
            "weight": 827941,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 18,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 32,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 33,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 22,
                    "weight": 121916,
                    "pos": 3
                },
                {
                    "id": 19,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 34,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 35,
                    "weight": 18559,
                    "pos": 6
                },
                {
                    "id": 23,
                    "weight": 121916,
                    "pos": 7
                },
                {
                    "id": 21,
                    "weight": 121916,
                    "pos": 8
                },
                {
                    "id": 20,
                    "weight": 121916,
                    "pos": 9
                },
                {
                    "id": 42,
                    "weight": 228923,
                    "pos": 10
                }
            ]
        },
        {
            "id": -14,
            "name": "FT1-NodeD~ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 340277,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 18,
                    "weight": 18559,
                    "pos": 0
                },
                {
                    "id": 32,
                    "weight": 18559,
                    "pos": 1
                },
                {
                    "id": 33,
                    "weight": 18559,
                    "pos": 2
                },
                {
                    "id": 19,
                    "weight": 18559,
                    "pos": 3
                },
                {
                    "id": 34,
                    "weight": 18559,
                    "pos": 4
                },
                {
                    "id": 35,
                    "weight": 18559,
                    "pos": 5
                },
                {
                    "id": 42,
                    "weight": 228923,
                    "pos": 6
                }
            ]
        },
        {
            "id": -15,
            "name": "FT1-NodeD~hdd",
            "type_id": 1,
            "type_name": "host",
            "weight": 487664,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": 22,
                    "weight": 121916,
                    "pos": 0
                },
                {
                    "id": 23,
                    "weight": 121916,
                    "pos": 1
                },
                {
                    "id": 21,
                    "weight": 121916,
                    "pos": 2
                },
                {
                    "id": 20,
                    "weight": 121916,
                    "pos": 3
                }
            ]
        },
        {
            "id": -16,
            "name": "FT1-NodeA~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -17,
            "name": "FT1-NodeB~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -18,
            "name": "FT1-NodeC~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -19,
            "name": "FT1-NodeD~old-ssd",
            "type_id": 1,
            "type_name": "host",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": []
        },
        {
            "id": -20,
            "name": "default~old-ssd",
            "type_id": 11,
            "type_name": "root",
            "weight": 0,
            "alg": "straw2",
            "hash": "rjenkins1",
            "items": [
                {
                    "id": -16,
                    "weight": 0,
                    "pos": 0
                },
                {
                    "id": -17,
                    "weight": 0,
                    "pos": 1
                },
                {
                    "id": -18,
                    "weight": 0,
                    "pos": 2
                },
                {
                    "id": -19,
                    "weight": 0,
                    "pos": 3
                }
            ]
        }
    ],
    "rules": [
        {
            "rule_id": 0,
            "rule_name": "ssd_rule",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -2,
                    "item_name": "default~ssd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        },
        {
            "rule_id": 1,
            "rule_name": "hdd_rule",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -6,
                    "item_name": "default~hdd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        },
        {
            "rule_id": 2,
            "rule_name": "old_nvme",
            "type": 1,
            "steps": [
                {
                    "op": "take",
                    "item": -20,
                    "item_name": "default~old-ssd"
                },
                {
                    "op": "chooseleaf_firstn",
                    "num": 0,
                    "type": "host"
                },
                {
                    "op": "emit"
                }
            ]
        }
    ],
    "tunables": {
        "choose_local_tries": 0,
        "choose_local_fallback_tries": 0,
        "choose_total_tries": 50,
        "chooseleaf_descend_once": 1,
        "chooseleaf_vary_r": 1,
        "chooseleaf_stable": 1,
        "straw_calc_version": 1,
        "allowed_bucket_algs": 54,
        "profile": "jewel",
        "optimal_tunables": 1,
        "legacy_tunables": 0,
        "minimum_required_version": "jewel",
        "require_feature_tunables": 1,
        "require_feature_tunables2": 1,
        "has_v2_rules": 0,
        "require_feature_tunables3": 1,
        "has_v3_rules": 0,
        "has_v4_buckets": 1,
        "require_feature_tunables5": 1,
        "has_v5_rules": 0
    },
    "choose_args": {}
}



It may help to manually edit the CRUSH map as detailed here:

https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/


At the end of the decompiled CRUSH map is a “tunables” section.  Look for the “c|hoose_total_tries|”
line, which should have the default value of 50.  Change that to 100.



On Nov 16, 2024, at 11:13 AM, Roland Giesler <rol...@giesler.za.net> wrote:

On 2024/11/15 16:37, Anthony D'Atri wrote:
Only 2 OSDs in the acting set.  This isn’t a size=2 pool is it?  Did you restart osd.39?

Restarted both repeatedly.  Has no effect.



On Nov 15, 2024, at 9:32 AM, Roland Giesler <rol...@giesler.za.net> wrote:

"acting": [
39,
1


_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to