Hi Nikhil (and also Torkil who wrote a similar suggestion to me directly)
Thanks for the suggestions I tried out adding the filter_logic and sizes to
see if i could nudge it to the correct setup, e.g.
---
placement:
host_pattern: "mimer-osd02"
service_id: osd_spec
service_type: osd
spec:
block_db_size: 50000000000
data_devices:
rotational: 1
db_devices:
rotational: 0
size: 50G:700G
filter_logic: AND
Just adding "filter_logic: AND" changed nothing; it still wants to only use
the DATA device. If i add the size and block_db_size, then it still
continues to ignore the available space on the NVME, and then just ignores
the specification and does this madness instead: (mpathau is the slow HDD):
################
OSDSPEC PREVIEWS
################
+---------+----------+-------------+---------------------+---------------------+-----+
|SERVICE |NAME |HOST |DATA |DB
|WAL |
+---------+----------+-------------+---------------------+---------------------+-----+
|osd |osd_spec |mimer-osd02 |/dev/mapper/mpathau
|/dev/mapper/mpathau |- |
+---------+----------+-------------+---------------------+---------------------+-----+
Yikes! That's terrible..
Maybe the issue that Robert linked is blocking the NVME from being used (I
am on ceph 19 here and the symptoms fit exactly what Robert described, if i
wipe the entire NVME it probably works (it did last i time i tried)), but
it's also quite frustrating that it so happily ignores the specification in
all these scenarios.
Specification is treated more like a recommendation
On Tue, Nov 4, 2025 at 5:51 PM Nikhil Mitra (nikmitra) <[email protected]>
wrote:
> We have had success using this spec format, not sure if this is helpful
> but we specify the block_db_size and use ceph-volume and LVM to achieve
> this.
>
> service_type: osd
> service_id: noncolocated_hdd
> service_name: osd.noncolocated_hdd
> spec:
> block_db_size: 250G
> crush_device_class: hdd
> data_devices:
> rotational: 1
> size: 5.46T
> db_devices:
> rotational: 0
> size: 3.49T
> filter_logic: AND
> objectstore: bluestore
>
> --
>
> Regards,
>
> Nikhil Mitra
>
>
> * Cisco Confidential From: *Mikael Öhman <[email protected]>
> *Date: *Tuesday, November 4, 2025 at 10:31 AM
> *To: *Robert Sander <[email protected]>
> *Cc: *[email protected] <[email protected]>
> *Subject: *[ceph-users] Re: HDD replacements with shared NVME lvm for DB
> - what am I missing here
>
> Thanks Robert, I will keep an eye on the issue
> I never actually had any success with this even back when running ceph 18
> either (actually, i think I saw the same with 17).
> It's very possible that I was simply doing something wrong back then.
> E.g. I just noticed that ceph orch device ls
> also won't find any of my NVMEs at all if they happen to be picked up by
> multipath.
> I.e. these won't even show up in ceph orch device ls at all (lsblk):
> nvme0n1
>
> 259:0
> 0 745.2G 0 disk
> └─mpathar
>
> 253:82
> 0 745.2G 0 mpath
>
>
> ├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28
> 253:88 0 53.2G 0 lvm
> ...
> but after i blacklist the nvmes from multipathd, they starts showing up in
> device ls (but still won't be used by orch due to the current issues):
> nvme0n1
>
> 259:0
> 0 745.2G 0 disk
>
> ├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179
> 253:29 0 53.2G 0 lvm
>
> Maybe that was what prevented the nvme from being picked up before 🤔
>
>
>
> On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <[email protected]
> >
> wrote:
>
> > Hi,
> >
> > Am 04.11.25 um 2:01 PM schrieb Mikael Öhman:
> > > My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs
> per
> > > NVME).
> > > It's all using ceph orch, containerized setup, using LVMs, so it's
> > probably
> > > the most conventional HDD based setup one can do.
> > >
> > > I have the basic osd spec:
> > > ---
> > > placement:
> > > host_pattern: "mimer-osd02"
> > > service_id: osd_spec
> > > service_type: osd
> > > spec:
> > > data_devices:
> > > rotational: 1
> > > db_devices:
> > > rotational: 0
> > >
> > > But unless the NVME is completely empty orch will just never pick it up
> > > (which I of course don't want to do, as it brings down 13 other OSDs).
> > > Instead orch just flat out ignores the requirement that db_devices must
> > go
> > > on rotational: 0 and incorrectly suggests the broken setup:
> >
> >
> > Ceph 19 has this bug:
> >
> > https://tracker.ceph.com/issues/72696
> >
> > Regards
> > --
> > Robert Sander
> > Linux Consultant
> >
> > Heinlein Consulting GmbH
> > Schwedter Str. 8/9b, 10119 Berlin
> >
> > https://www.heinlein-support.de
> >
> > Tel: +49 30 405051 - 0
> > Fax: +49 30 405051 - 19
> >
> > Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> > Geschäftsführer: Peer Heinlein - Sitz: Berlin
> > _______________________________________________
> > ceph-users mailing list -- [email protected]
> > To unsubscribe send an email to [email protected]
> >
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
