We have had success using this spec format, not sure if this is helpful but we
specify the block_db_size and use ceph-volume and LVM to achieve this.
service_type: osd
service_id: noncolocated_hdd
service_name: osd.noncolocated_hdd
spec:
block_db_size: 250G
crush_device_class: hdd
data_devices:
rotational: 1
size: 5.46T
db_devices:
rotational: 0
size: 3.49T
filter_logic: AND
objectstore: bluestore
--
Regards,
Nikhil Mitra
Cisco Confidential
From: Mikael Öhman <[email protected]>
Date: Tuesday, November 4, 2025 at 10:31 AM
To: Robert Sander <[email protected]>
Cc: [email protected] <[email protected]>
Subject: [ceph-users] Re: HDD replacements with shared NVME lvm for DB - what
am I missing here
Thanks Robert, I will keep an eye on the issue
I never actually had any success with this even back when running ceph 18
either (actually, i think I saw the same with 17).
It's very possible that I was simply doing something wrong back then.
E.g. I just noticed that ceph orch device ls
also won't find any of my NVMEs at all if they happen to be picked up by
multipath.
I.e. these won't even show up in ceph orch device ls at all (lsblk):
nvme0n1
259:0
0 745.2G 0 disk
└─mpathar
253:82
0 745.2G 0 mpath
├─ceph--1b309b1e--a4a6--4861--b16c--7c06ecde1a3d-osd--db--82b8fd01--ede3--4c37--a145--f2bea1cf8f28
253:88 0 53.2G 0 lvm
...
but after i blacklist the nvmes from multipathd, they starts showing up in
device ls (but still won't be used by orch due to the current issues):
nvme0n1
259:0
0 745.2G 0 disk
├─ceph--419bd62d--8cd0--4ff1--9b29--f87c9c6aae8b-osd--db--a9967578--3127--4506--a8b8--182621b4f179
253:29 0 53.2G 0 lvm
Maybe that was what prevented the nvme from being picked up before 🤔
On Tue, Nov 4, 2025 at 2:38 PM Robert Sander <[email protected]>
wrote:
> Hi,
>
> Am 04.11.25 um 2:01 PM schrieb Mikael Öhman:
> > My hosts have 42 HDDs, sharing 3 NVMEs for DB/WAL partitions (14 OSDs per
> > NVME).
> > It's all using ceph orch, containerized setup, using LVMs, so it's
> probably
> > the most conventional HDD based setup one can do.
> >
> > I have the basic osd spec:
> > ---
> > placement:
> > host_pattern: "mimer-osd02"
> > service_id: osd_spec
> > service_type: osd
> > spec:
> > data_devices:
> > rotational: 1
> > db_devices:
> > rotational: 0
> >
> > But unless the NVME is completely empty orch will just never pick it up
> > (which I of course don't want to do, as it brings down 13 other OSDs).
> > Instead orch just flat out ignores the requirement that db_devices must
> go
> > on rotational: 0 and incorrectly suggests the broken setup:
>
>
> Ceph 19 has this bug:
>
> https://tracker.ceph.com/issues/72696
>
> Regards
> --
> Robert Sander
> Linux Consultant
>
> Heinlein Consulting GmbH
> Schwedter Str. 8/9b, 10119 Berlin
>
> https://www.heinlein-support.de
>
> Tel: +49 30 405051 - 0
> Fax: +49 30 405051 - 19
>
> Amtsgericht Berlin-Charlottenburg - HRB 220009 B
> Geschäftsführer: Peer Heinlein - Sitz: Berlin
> _______________________________________________
> ceph-users mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
