Hi Marco,
the following log line (unfortunately it was cut off) sheds some light:
"
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 -1
bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes bluestore_cache_meta_>
"
Likely it says that sum of bluestore_cache_meta_ratio +
bluestore_cache_kv_ratio + bluestore_cache_kv_onode_ratio config
parameters exceeds 1.0
So one has to tune the parameters in a way to get the sum less or equal
to 1.0.
Default settings are:
bluestore_cache_meta_ratio = 0.45
bluestore_cache_kv_ratio = 0.45
bluestore_cache_kv_onode_ratio = 0.04
Thanks,
Igor
On 29.04.2025 13:36, Marco Pizzolo wrote:
Hello Everyone,
I'm upgrading from 18.2.4 to 18.2.6, and I have a 4-node cluster with 8
NVMe's per node. Each NVMe is split into 2 OSDs. The upgrade went through
the mgr, mon, crash and began upgrading OSDs.
The OSDs it was upgrading were not coming back online.
I tried rebooting, and no luck.
journalctl -xe shows the following:
░░ The unit
docker-02cb79ef9a657cdaa26b781966aa6d2f1d5e54cdc9efa6c5ff1f0e98c3a866e4.scope
has successfully entered the 'dead' state.
Apr 29 06:24:09 prdhcistonode01 dockerd[2967]:
time="2025-04-29T06:24:09.282073583-04:00" level=info msg="ignoring event"
container=76c56ddd668015de0022bfa2527060e64a9513>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.282129114-04:00" level=info msg="shim
disconnected" id=76c56ddd668015de0022bfa2527060e64a95137>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.282219664-04:00" level=warning msg="cleaning up
after shim disconnected" id=76c56ddd668015de00>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.282242484-04:00" level=info msg="cleaning up dead
shim"
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 1 mClockScheduler:
set_osd_capacity_params_from_config: osd_bandwidth_cost_p>
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 0 osd.3:0.OSDShard using op
scheduler mclock_scheduler, cutoff=196
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000
/var/lib/ceph/osd/ceph-3/block) open path /var/lib/cep>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.292047607-04:00" level=warning msg="cleanup
warnings time=\"2025-04-29T06:24:09-04:00\" level=>
Apr 29 06:24:09 prdhcistonode01 dockerd[2967]:
time="2025-04-29T06:24:09.292163618-04:00" level=info msg="ignoring event"
container=02cb79ef9a657cdaa26b781966aa6d2f1d5e54>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.292216428-04:00" level=info msg="shim
disconnected" id=02cb79ef9a657cdaa26b781966aa6d2f1d5e54c>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.292277279-04:00" level=warning msg="cleaning up
after shim disconnected" id=02cb79ef9a657cdaa2>
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.292291949-04:00" level=info msg="cleaning up dead
shim"
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000
/var/lib/ceph/osd/ceph-3/block) open size 640122932428>
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 -1
bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes bluestore_cache_meta_>
Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug
2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000
/var/lib/ceph/osd/ceph-3/block) close
Apr 29 06:24:09 prdhcistonode01 containerd[2797]:
time="2025-04-29T06:24:09.303385220-04:00" level=warning msg="cleanup
warnings time=\"2025-04-29T06:24:09-04:00\" level=>
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 1 mClockScheduler:
set_osd_capacity_params_from_config: osd_bandwidth_cost_p>
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 0 osd.0:0.OSDShard using op
scheduler mclock_scheduler, cutoff=196
Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug
2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 osd.15 0 OSD:init: unable to
mount object store
Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug
2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 ** ERROR: osd init failed:
(22) Invalid argument
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/cep>
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000
/var/lib/ceph/osd/ceph-0/block) open size 640122932428>
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 -1
bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes bluestore_cache_meta_>
Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug
2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000
/var/lib/ceph/osd/ceph-0/block) close
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 1 mClockScheduler:
set_osd_capacity_params_from_config: osd_bandwidth_cost_p>
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 0 osd.8:0.OSDShard using op
scheduler mclock_scheduler, cutoff=196
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000
/var/lib/ceph/osd/ceph-8/block) open path /var/lib/cep>
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000
/var/lib/ceph/osd/ceph-8/block) open size 640122932428>
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 -1
bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes bluestore_cache_meta_>
Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug
2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000
/var/lib/ceph/osd/ceph-8/block) close
Apr 29 06:24:09 prdhcistonode01 systemd[1]:
ceph-fbc38f5c-a3a6-11ea-805c-3b954db9ce7a@osd.12.service: Main process
exited, code=exited, status=1/FAILURE
Any help you can offer would be greatly appreciated. This is running in
docker:
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:08:01 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:08:01 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Thanks,
Marco
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
