Is the NVMe (DB) drive also have other data on it ? Or is it an empty drive dedicated for the DB ?
What does ceph orch device ls <hostname> output give you for that device ? -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <[email protected]> Date: Tuesday, November 4, 2025 at 11:57 AM To: Nikhil Mitra (nikmitra) <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Hi Nikhil (and also Torkil who wrote a similar suggestion to me directly) Thanks for the suggestions I tried out adding the filter_logic and sizes to see if i could nudge it to the correct setup, e.g. --- placement: host_pattern: "mimer-osd02" service_id: osd_spec service_type: osd spec: block_db_size: 50000000000 data_devices: rotational: 1 db_devices: rotational: 0 size: 50G:700G filter_logic: AND Just adding "filter_logic: AND" changed nothing; it still wants to only use the DATA device. If i add the size and block_db_size, then it still continues to ignore the available space on the NVME, and then just ignores the specification and does this madness instead: (mpathau is the slow HDD): ################ OSDSPEC PREVIEWS ################ +---------+----------+-------------+---------------------+---------------------+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+----------+-------------+---------------------+---------------------+-----+ |osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau |/dev/mapper/mpathau |- | +---------+----------+-------------+---------------------+---------------------+-----+ Yikes! That's terrible.. Maybe the issue that Robert linked is blocking the NVME from being used (I am on ceph 19 here and the symptoms fit exactly what Robert described, if i wipe the entire NVME it probably works (it did last i time i tried)), but it's also quite frustrating that it so happily ignores the specification in all these scenarios. Specification is treated more like a recommendation On Tue, Nov 4, 2025 at 5:51 PM Nikhil Mitra (nikmitra) <[email protected]<mailto:[email protected]>> wrote: We have had success using this spec format, not sure if this is helpful but we specify the block_db_size and use ceph-volume and LVM to achieve this. service_type: osd service_id: noncolocated_hdd service_name: osd.noncolocated_hdd spec: block_db_size: 250G crush_device_class: hdd data_devices: rotational: 1 size: 5.46T db_devices: rotational: 0 size: 3.49T filter_logic: AND objectstore: bluestore -- Regards, Nikhil Mitra Cisco Confidential From: Mikael Öhman <[email protected]<mailto:[email protected]>> Date: Tuesday, November 4, 2025 at 10:31 AM To: Robert Sander <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what am I missing here Thanks Robert, I will keep an eye on the issue I never actually had any success with this even back when running ceph 18 either (actually, i think I saw the same with 17). It's very possible that I was simply doing something wrong back then. E.g. I just noticed that ceph orch device ls also won't find any of my NVMEs at all if they happen to be picked up by multipath. I.e. these won't even show up in ceph orch device ls at all (lsblk): nvme0n1 259:0 0 745.2G 0 disk └─mpathar 253:82 0 745.2G 0 mpath ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28 253:88 0 53.2G 0 lvm ... but after i blacklist the nvmes from multipathd, they starts showing up in device ls (but still won't be used by orch due to the current issues): nvme0n1 259:0 0 745.2G 0 disk ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179 253:29 0 53.2G 0 lvm Maybe that was what prevented the nvme from being picked up before 🤔 On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <[email protected]<mailto:[email protected]>> wrote: > Hi, > > Am 04.11.25 um 2:01 PM schrieb Mikael Öhman: > > My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per > > NVME). > > It's all using ceph orch, containerized setup, using LVMs, so it's > probably > > the most conventional HDD based setup one can do. > > > > I have the basic osd spec: > > --- > > placement: > > host_pattern: "mimer-osd02" > > service_id: osd_spec > > service_type: osd > > spec: > > data_devices: > > rotational: 1 > > db_devices: > > rotational: 0 > > > > But unless the NVME is completely empty orch will just never pick it up > > (which I of course don't want to do, as it brings down 13 other OSDs). > > Instead orch just flat out ignores the requirement that db_devices must > go > > on rotational: 0 and incorrectly suggests the broken setup: > > > Ceph 19 has this bug: > > https://tracker.ceph.com/issues/72696 > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- [email protected]<mailto:[email protected]> > To unsubscribe send an email to > [email protected]<mailto:[email protected]> > _______________________________________________ ceph-users mailing list -- [email protected]<mailto:[email protected]> To unsubscribe send an email to [email protected]<mailto:[email protected]> _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
