Not a unique issue and I suspect that it affects lots of people who don’t know
yet.
Might be that you should rm the old LVM first or specify it with an explicit
create command.
> On Sep 27, 2024, at 8:55 AM, mailing-lists <mailing-li...@indane.de> wrote:
>
> Dear Ceph-users,
> I have a problem that I'd like to have your input for.
>
> Preface:
> I have got a test-cluster and a productive-cluster. Both are setup the same
> and both are having the same "issue". I am running Ubuntu 22.04 and deployed
> ceph 17.2.3 via cephadm. Upgraded to 17.2.7 later on, which is the version we
> are currently running. Since the issue seem to be the exact same on the
> test-cluster, I will post test-cluster-outputs here for better readability.
>
> The issue:
> I have replaced disks and after the replacement, it does not show that it
> would use the NVMe as WAL device anymore. The LV still exists, but the
> metadata of the osd does not show it, as it would be with any other osd/hdd,
> that hasnt been replaced.
>
> ODS.1 (incorrect, bluefs_dedicated_wal: "0")
> ```
> {
> "id": 1,
> "arch": "x86_64",
> "back_addr":
> "[v2:192.168.6.241:6802/3213655489,v1:192.168.6.241:6803/3213655489]",
> "back_iface": "",
> "bluefs": "1",
> "bluefs_dedicated_db": "0",
> "bluefs_dedicated_wal": "0",
> "bluefs_single_shared_device": "1",
> "bluestore_bdev_access_mode": "blk",
> "bluestore_bdev_block_size": "4096",
> "bluestore_bdev_dev_node": "/dev/dm-3",
> "bluestore_bdev_devices": "sdd",
> "bluestore_bdev_driver": "KernelDevice",
> "bluestore_bdev_optimal_io_size": "0",
> "bluestore_bdev_partition_path": "/dev/dm-3",
> "bluestore_bdev_rotational": "1",
> "bluestore_bdev_size": "17175674880",
> "bluestore_bdev_support_discard": "1",
> "bluestore_bdev_type": "hdd",
> "bluestore_min_alloc_size": "4096",
> "ceph_release": "quincy",
> "ceph_version": "ceph version 17.2.7
> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)",
> "ceph_version_short": "17.2.7",
> "ceph_version_when_created": "",
> "container_hostname": "bi-ubu-srv-ceph2-01",
> "container_image":
> "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b",
> "cpu": "AMD EPYC 75F3 32-Core Processor",
> "created_at": "",
> "default_device_class": "hdd",
> "device_ids": "sdd=QEMU_HARDDISK_drive-scsi3",
> "device_paths": "sdd=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:3:0",
> "devices": "sdd",
> "distro": "centos",
> "distro_description": "CentOS Stream 8",
> "distro_version": "8",
> "front_addr":
> "[v2:<my_public_ip>.241:6800/3213655489,v1:<my_public_ip>.241:6801/3213655489]",
> "front_iface": "",
> "hb_back_addr":
> "[v2:192.168.6.241:6806/3213655489,v1:192.168.6.241:6807/3213655489]",
> "hb_front_addr":
> "[v2:<my_public_ip>.241:6804/3213655489,v1:<my_public_ip>.241:6805/3213655489]",
> "hostname": "bi-ubu-srv-ceph2-01",
> "journal_rotational": "1",
> "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024",
> "kernel_version": "5.15.0-122-generic",
> "mem_swap_kb": "4018172",
> "mem_total_kb": "5025288",
> "network_numa_unknown_ifaces": "back_iface,front_iface",
> "objectstore_numa_unknown_devices": "sdd",
> "os": "Linux",
> "osd_data": "/var/lib/ceph/osd/ceph-1",
> "osd_objectstore": "bluestore",
> "osdspec_affinity": "dashboard-admin-1661853488642",
> "rotational": "1"
> }
> ```
>
> ODS.0 (correct, bluefs_dedicated_wal: "1")
> ```
> {
> "id": 0,
> "arch": "x86_64",
> "back_addr":
> "[v2:192.168.6.241:6810/3249286142,v1:192.168.6.241:6811/3249286142]",
> "back_iface": "",
> "bluefs": "1",
> "bluefs_dedicated_db": "0",
> "bluefs_dedicated_wal": "1",
> "bluefs_single_shared_device": "0",
> "bluefs_wal_access_mode": "blk",
> "bluefs_wal_block_size": "4096",
> "bluefs_wal_dev_node": "/dev/dm-0",
> "bluefs_wal_devices": "sdb",
> "bluefs_wal_driver": "KernelDevice",
> "bluefs_wal_optimal_io_size": "0",
> "bluefs_wal_partition_path": "/dev/dm-0",
> "bluefs_wal_rotational": "0",
> "bluefs_wal_size": "4290772992",
> "bluefs_wal_support_discard": "1",
> "bluefs_wal_type": "ssd",
> "bluestore_bdev_access_mode": "blk",
> "bluestore_bdev_block_size": "4096",
> "bluestore_bdev_dev_node": "/dev/dm-2",
> "bluestore_bdev_devices": "sdc",
> "bluestore_bdev_driver": "KernelDevice",
> "bluestore_bdev_optimal_io_size": "0",
> "bluestore_bdev_partition_path": "/dev/dm-2",
> "bluestore_bdev_rotational": "1",
> "bluestore_bdev_size": "17175674880",
> "bluestore_bdev_support_discard": "1",
> "bluestore_bdev_type": "hdd",
> "bluestore_min_alloc_size": "4096",
> "ceph_release": "quincy",
> "ceph_version": "ceph version 17.2.7
> (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)",
> "ceph_version_short": "17.2.7",
> "ceph_version_when_created": "",
> "container_hostname": "bi-ubu-srv-ceph2-01",
> "container_image":
> "quay.io/ceph/ceph@sha256:28323e41a7d17db238bdcc0a4d7f38d272f75c1a499bc30f59b0b504af132c6b",
> "cpu": "AMD EPYC 75F3 32-Core Processor",
> "created_at": "",
> "default_device_class": "hdd",
> "device_ids":
> "sdb=QEMU_HARDDISK_drive-scsi1,sdc=QEMU_HARDDISK_drive-scsi2",
> "device_paths":
> "sdb=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:1:0,sdc=/dev/disk/by-path/pci-0000:00:05.0-scsi-0:0:2:0",
> "devices": "sdb,sdc",
> "distro": "centos",
> "distro_description": "CentOS Stream 8",
> "distro_version": "8",
> "front_addr":
> "[v2:<my_public_ip>.241:6808/3249286142,v1:<my_public_ip>.241:6809/3249286142]",
> "front_iface": "",
> "hb_back_addr":
> "[v2:192.168.6.241:6814/3249286142,v1:192.168.6.241:6815/3249286142]",
> "hb_front_addr":
> "[v2:<my_public_ip>.241:6812/3249286142,v1:<my_public_ip>.241:6813/3249286142]",
> "hostname": "bi-ubu-srv-ceph2-01",
> "journal_rotational": "0",
> "kernel_description": "#132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024",
> "kernel_version": "5.15.0-122-generic",
> "mem_swap_kb": "4018172",
> "mem_total_kb": "5025288",
> "network_numa_unknown_ifaces": "back_iface,front_iface",
> "objectstore_numa_unknown_devices": "sdb,sdc",
> "os": "Linux",
> "osd_data": "/var/lib/ceph/osd/ceph-0",
> "osd_objectstore": "bluestore",
> "osdspec_affinity": "dashboard-admin-1661853488642",
> "rotational": "1"
> }
> ```
>
> The LV that should contain the WAL is existing:
> ```
> :/# lvs
> LV VG Attr LSize Pool Origin
> Data% Meta% Move Log Cpy%Sync Convert
> osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc
> ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368 -wi-ao---- <16.00g
> osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f
> ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35 -wi-ao---- <16.00g
> osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd
> ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-ao---- <4.00g
> osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431
> ceph-a84cbc55-33b5-4c31-8426-66e375fc157b -wi-a----- <4.00g
> ```
> We can also see there, that there is no (o) as in open in the LV attributes.
>
> For good measure, this is the pattern for the orch osd service.
> ```
> service_type: osd
> service_id: dashboard-admin-1661853488642
> service_name: osd.dashboard-admin-1661853488642
> placement:
> host_pattern: '*'
> spec:
> data_devices:
> size: 16GB
> db_devices:
> rotational: false
> filter_logic: AND
> objectstore: bluestore
> wal_devices:
> rotational: false
> status:
> created: '2022-08-30T09:58:11.210571Z'
> last_refresh: '2024-09-27T12:19:14.619434Z'
> running: 16
> size: 16
> ```
>
> "ceph-volume lvm list" looks fine, in my opinion, as it should be:
> ```
> {
> "0": [
> {
> "devices": [
> "/dev/sdc"
> ],
> "lv_name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "lv_path":
> "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "lv_size": "17175674880",
> "lv_tags":
> "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et",
> "lv_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk",
> "name": "osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "path":
> "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "tags": {
> "ceph.block_device":
> "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk",
> "ceph.cephx_lockbox_secret": "",
> "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a",
> "ceph.cluster_name": "ceph",
> "ceph.crush_device_class": "",
> "ceph.encrypted": "0",
> "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc",
> "ceph.osd_id": "0",
> "ceph.osdspec_affinity": "dashboard-admin-1661853488642",
> "ceph.type": "block",
> "ceph.vdo": "0",
> "ceph.wal_device":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et"
> },
> "type": "block",
> "vg_name": "ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368"
> },
> {
> "devices": [
> "/dev/sdb"
> ],
> "lv_name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "lv_path":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "lv_size": "4290772992",
> "lv_tags":
> "ceph.block_device=/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.block_uuid=3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d472bf9f-c17d-4939-baf5-514a07db66bc,ceph.osd_id=0,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd,ceph.wal_uuid=BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et",
> "lv_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et",
> "name": "osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "path":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "tags": {
> "ceph.block_device":
> "/dev/ceph-7cd212d6-29af-4c14-8e0b-bcd1984ba368/osd-block-d472bf9f-c17d-4939-baf5-514a07db66bc",
> "ceph.block_uuid": "3tkcvo-XKmb-03UQ-xMie-1Y1d-6ptW-NugdPk",
> "ceph.cephx_lockbox_secret": "",
> "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a",
> "ceph.cluster_name": "ceph",
> "ceph.crush_device_class": "",
> "ceph.encrypted": "0",
> "ceph.osd_fsid": "d472bf9f-c17d-4939-baf5-514a07db66bc",
> "ceph.osd_id": "0",
> "ceph.osdspec_affinity": "dashboard-admin-1661853488642",
> "ceph.type": "wal",
> "ceph.vdo": "0",
> "ceph.wal_device":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-3e18feca-01cf-4c5b-9934-8a30bc242bbd",
> "ceph.wal_uuid": "BgRlZX-YX36-SM0q-UiLE-B5xC-eBbO-yxL3et"
> },
> "type": "wal",
> "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b"
> }
> ],
> "1": [
> {
> "devices": [
> "/dev/sdd"
> ],
> "lv_name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "lv_path":
> "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "lv_size": "17175674880",
> "lv_tags":
> "ceph.block_device=/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.block_uuid=QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=d94bda82-e59f-4d3d-81cd-28ea69c5e02f,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=block,ceph.vdo=0",
> "lv_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO",
> "name": "osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "path":
> "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "tags": {
> "ceph.block_device":
> "/dev/ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35/osd-block-d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "ceph.block_uuid": "QdBLn0-aWXn-Ud7g-k7E3-91Yv-HqqJ-Xh3TuO",
> "ceph.cephx_lockbox_secret": "",
> "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a",
> "ceph.cluster_name": "ceph",
> "ceph.crush_device_class": "",
> "ceph.encrypted": "0",
> "ceph.osd_fsid": "d94bda82-e59f-4d3d-81cd-28ea69c5e02f",
> "ceph.osd_id": "1",
> "ceph.osdspec_affinity": "dashboard-admin-1661853488642",
> "ceph.type": "block",
> "ceph.vdo": "0"
> },
> "type": "block",
> "vg_name": "ceph-960d0016-077a-4ffa-a60e-32f6efcfcb35"
> },
> {
> "devices": [
> "/dev/sdb"
> ],
> "lv_name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431",
> "lv_path":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431",
> "lv_size": "4290772992",
> "lv_tags":
> "ceph.block_device=/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.block_uuid=gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ,ceph.cephx_lockbox_secret=,ceph.cluster_fsid=a12b3ade-2849-11ed-9b46-c5b62beb178a,ceph.cluster_name=ceph,ceph.crush_device_class=,ceph.encrypted=0,ceph.osd_fsid=7a1d0007-71ff-4011-8a18-e6de1499cbdf,ceph.osd_id=1,ceph.osdspec_affinity=dashboard-admin-1661853488642,ceph.type=wal,ceph.vdo=0,ceph.wal_device=/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431,ceph.wal_uuid=UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq",
> "lv_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq",
> "name": "osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431",
> "path":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431",
> "tags": {
> "ceph.block_device":
> "/dev/ceph-179656b5-6783-4b4e-bb7e-695d4523072f/osd-block-7a1d0007-71ff-4011-8a18-e6de1499cbdf",
> "ceph.block_uuid": "gDt63k-v0Qd-89WE-bOGO-cmZv-Sc8v-tYOagJ",
> "ceph.cephx_lockbox_secret": "",
> "ceph.cluster_fsid": "a12b3ade-2849-11ed-9b46-c5b62beb178a",
> "ceph.cluster_name": "ceph",
> "ceph.crush_device_class": "",
> "ceph.encrypted": "0",
> "ceph.osd_fsid": "7a1d0007-71ff-4011-8a18-e6de1499cbdf",
> "ceph.osd_id": "1",
> "ceph.osdspec_affinity": "dashboard-admin-1661853488642",
> "ceph.type": "wal",
> "ceph.vdo": "0",
> "ceph.wal_device":
> "/dev/ceph-a84cbc55-33b5-4c31-8426-66e375fc157b/osd-wal-a6202ebe-0959-40c6-8ba2-bc387d550431",
> "ceph.wal_uuid": "UOffkj-TBZJ-9b66-wIW7-3DhH-xibs-Y5t5Xq"
> },
> "type": "wal",
> "vg_name": "ceph-a84cbc55-33b5-4c31-8426-66e375fc157b"
> }
> ]
> }
> ```
> We can see two devices for osd.1, but the osd does not seem to use the WAL
> device.
>
> How can i tell osd.1 to use its WAL again?
> Why does the orch osd service not do it anyways?
>
> When i was replacing the disks i was using
> ```
> ceph orch osd rm 1 --replace
> ```
> The osd got back up and refilled, but without WAL.
>
>
>
>
> While I was diggin into this, i realized that the DB is not on the NVME, but
> on the HDD itself. Even though, i was specifying it in the pattern above:
> db_devices:
> rotational: false
>
> Shouldnt the db then be located on the NVMe/SSD where the wal is at? At least
> that is what the dashboard suggest, when creating OSDs. Of course I chose the
> NVME/SSD for DB and WAL.
>
>
> I hope this is somewhat readable and coherent.
>
> Thank you in advance!
>
> Best wishes and a nice weekend!
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
