Greetings community,
we have a setup comprising of 6 servers hosting CentOS 8 Minimal Installation
with CEPH Quincy version 18.2.2 supported by 20Gbps fiber optics NICs and a
dual Xeon Intel processors, bootstrapped the installation on the first node
then expanded to the others using the cephadm method, having the monitor
services deployed on 5 of these nodes as well as 3 manager nodes. Each server
has an NVMe boot disk as well as a 1TBs SATA SSD over which the OSDs are
deployed. An EC profile was created with k=3 and m=3, serving a CephFS
filesystem on top with NFS exports to serve other servers. Up to this point,
the setup is quite stable in the sense that upon emergency reboot or network
connection failure the OSDs did not fail and remain functional/started normally
after reboot.
At a certain point in our project, we had the need to activate the multipathd
service, adding the boot drive partition and the CEPH SSD to its blacklist as
to not be initialized for use by an mpath partition, the blacklist goes like so:
boot blacklist:
===============
blacklist {
wwid "eui.<drive_id>"
}
SATA SSD blacklist:
===================
blacklist {
wwid "naa.<drive_id>"
}
The above blacklist configuration ensures that both the boot disk as well as
CEPH's OSD function properly, with the following being lsblk output:
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda
8:0 0 894.3G 0 disk
└─ceph--<id>-osd--block--<block_id>
252:3 0 894.3G 0 lvm
nvme0n1
259:0 0 238.5G 0 disk
├─nvme0n1p1
259:1 0 600M 0 part /boot/efi
├─nvme0n1p2
259:2 0 1G 0 part /boot
└─nvme0n1p3
259:3 0 236.9G 0 part
├─centos-root
252:0 0 170G 0 lvm /
├─centos-swap
252:1 0 23.4G 0 lvm [SWAP]
├─centos-var_log_audit
252:2 0 7.5G 0 lvm /var/log/audit
├─centos-home
252:4 0 26G 0 lvm /home
└─centos-var_log
252:5 0 10G 0 lvm /var/log
In addition to the above multipathd configuration, we have use_devicesfile=1 in
/etc/lvm/lvm.conf, with /etc/lvm/devices/system.devices file being like so,
with PVID used from the output of the pvdisplay command, and the IDNAME value
extracted from the ouput of "ls -lha /dev/disk/by-id":
VERSION=1.1.1
IDTYPE=sys_wwid IDNAME=eui.<drive_id> DEVNAME=/dev/nvme0n1p3 PVID=<pvid> PART=3
IDTYPE=sys_wwid IDNAME=naa.<drive_id> DEVNAME=/dev/sda PVID=<pvid>
Issues started when performing certain tests regarding the system's integrity,
most important of which is emergency shutdown's and reboot of all the nodes,
the behavior that follows is that the OSDs are not started automatically as
well as their respective LVM volumes not properly showing (except on a single
node for some reason), hence the lsblk ouput changes like the snippet below,
requiring us rebooting the nodes one by one until all the OSDs are back online:
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda
8:0 0 894.3G 0 disk
nvme0n1
259:0 0 238.5G 0 disk
├─nvme0n1p1
259:1 0 600M 0 part /boot/efi
├─nvme0n1p2
259:2 0 1G 0 part /boot
└─nvme0n1p3
259:3 0 236.9G 0 part
├─centos-root
252:0 0 170G 0 lvm /
├─centos-swap
252:1 0 23.4G 0 lvm [SWAP]
├─centos-var_log_audit
252:2 0 7.5G 0 lvm /var/log/audit
├─centos-home
252:4 0 26G 0 lvm /home
└─centos-var_log
252:5 0 10G 0 lvm /var/log
Without the LVM configuration and the multipathd service enabled, everything
works fine, this behavior started happening after the changes. Attempting to
manually restart the OSDs from a manager node using ceph orch daemon restart
osd.n results in an error state, and even when manually starting the OSD on
each node via the bash /var/lib/ceph/<fsid>/osd.0/unit.run we receive the
following error:
--> Failed to activate via raw: did not find any matching OSD to activate
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir
--dev /dev/ceph-<id>/osd-block-<block_id> --path /var/lib/ceph/osd/ceph-0
--no-mon-config
stderr: failed to read label for /dev/ceph-<id>/osd-block-<block_id>: (2) No
such file or directory
2024-03-30T12:42:54.014+0000 7f845296a980 -1
bluestore(/dev/ceph-<id>/osd-block-<block_id>) _read_bdev_label failed to open
/dev/ceph-<id>/osd-block-<block_id>: (2) No such file or directory
--> Failed to activate via LVM: command returned non-zero exit status: 1
--> Failed to activate via simple: 'Namespace' object has no attribute
'json_config'
--> Failed to activate any OSD(s)
A successful run of the same command but with success results in the following
output:
/bin/bash /var/lib/ceph/<fsid>/osd.0/unit.run
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path
/var/lib/ceph/osd/ceph-0 --no-mon-config --dev
/dev/mapper/ceph--<id>-osd--block--<block_id>
Running command: /usr/bin/chown -h ceph:ceph
/dev/mapper/ceph--<id>-osd--block--<block_id>
Running command: /usr/bin/chown -R ceph:ceph /dev/dm-5
Running command: /usr/bin/ln -s /dev/mapper/ceph--<id>-osd--block--<block_id>
/var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
--> ceph-volume raw activate successful for osd ID: 0
ceph-<fsid>-osd-0
4361e2f166bcdeee6e9020dcbb153d3d7eec04e71d5b0b250440d4a3a0833f2c
It seems to us as if the logical volume in cases of failure is not even
detected at boot by the device-mapper, which is weird, it's also not showing in
the output of the dmsetup ls command in cases of failure. What could we be
missing here? What seems to be the conflict between CEPH OSDs and the
multipathd service or even the LVM configuration? Should the system.devices
entry be different than what we set? Is the multipathd blacklisting
configuration missing something? We have been working on trial and error
experiments for more than a week now and looked at the lvm2 as well as
multipathd logs (we could provide them upon request) but to no avail as nothing
indicates any errors, just normal logs with the difference being the missing
CEPH OSD LVM volume.
Best regards
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
