[ceph-users] Ceph-ansible: add a new HDD to an already provisioned WAL device
Hello all, we’ve set up a new Ceph cluster with a number of nodes which are all identically configured. There is one device vda which should act as WAL device for all other devices. Additionally, there are four other devices vdb, vdc, vdd, vde which use vda as WAL. The whole cluster was set up using ceph-ansible (branch stable-7.0) and Ceph version 17.2.0. Device configuration in osds.yml looks as follows: devices: [/dev/vdb, /dev/vdc, /dev/vdd, /dev/vde] bluestore_wal_devices: [/dev/vda] As expected vda contains four logical volumes for WAL each 1/4 of the overall vda disk size (‘ceph-ansible/group_vars/all.yml’ has default ‘block_db_size: -1’). After the initial setup, we’ve added an additional device vdf which should become a new OSD. The new OSD should use vda for WAL as well. This means the previous four WAL LVs have to be resized down to 1/5 and a new LV has to be added. Is it possible to retroactively add a new device to an already provisioned WAL device? We suspect that this is not possible because the ceph-bluestore-tool does not provide any way to shrink an existing BlueFS device. Only expanding is currently possible (https://docs.ceph.com/en/quincy/man/8/ceph-bluestore-tool/). Simply adding the new device to the devices list and rerunning the playbook does nothing. And so does only setting “devices: [/dev/vdf]” and “bluestore_wal_devices: [/dev/vda]”. In both cases vda is rejected because “Insufficient space (<10 extents) on vgs” which makes sense because vda is already fully used by the previous four OSD WALs. Thanks for the help and kind regards. Additional notes: - We’re testing pre-production on an emulated cluster hence the device names vdx and unusually small device sizes. - The output of `lsblk` after the initial setup looks as follows: ``` vda 252:008G 0 disk ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--3677c354--8d7d--4db9--a2b7--68aeb8248d40 253:202G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--52d71122--b573--4077--9633--968c178612fd 253:402G 0 lvm ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--2d7eb467--cfb1--4a00--8a45--273932036599 253:602G 0 lvm └─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--d7b13b79--219c--4002--9e92--370dff7a5376 253:802G 0 lvm vdb 252:16 08G 0 disk └─ceph--49ddaa8b--5d8f--4267--85f9--5cac608ce53d-osd--block--861a53c7--ee57--4c5f--9546--1dd7cb0185ef 253:108G 0 lvm vdc 252:32 05G 0 disk └─ceph--1ed9ee91--e071--4ea4--9703--d56d84d9ae0a-osd--block--8aacb66a--e29b--4b7a--8ad5--a9fb1f81c6d6 253:305G 0 lvm vdd 252:48 05G 0 disk └─ceph--554cdd8b--e722--41a9--8f64--c09c857cc0dc-osd--block--4dee3e1b--b50d--4154--b2ff--80cadb67e2a0 253:505G 0 lvm vde 252:64 05G 0 disk └─ceph--5d58de32--ca55--4895--8ac7--af94ee07672e-osd--block--3f563f40--0c1e--4cca--9325--d9534cceb711 253:705G 0 lvm vdf 252:80 05G 0 disk ``` - Ceph status is happy and healthy: ``` cluster: id: ff043ce8----e98d073c9d09 health: HEALTH_WARN mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum baloo-1,baloo-2,baloo-3 (age 13m) mgr: baloo-2(active, since 5m), standbys: baloo-3, baloo-1 mds: 1/1 daemons up, 1 standby osd: 24 osds: 24 up (since 4m), 24 in (since 5m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 7 pools, 177 pgs objects: 213 objects, 584 KiB usage: 98 MiB used, 138 GiB / 138 GiB avail pgs: 177 active+clean ``` ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
[ceph-users] Re: Ceph-ansible: add a new HDD to an already provisioned WAL device
``` > restart the osd: > ``` > [root@osd0 ~]# systemctl start ceph-osd@0 > ``` > remove tmp lv/vg/pv: > ``` > [root@osd0 ~]# lvremove /dev/vg_db_tmp/db-sdb > Do you really want to remove active logical volume vg_db_tmp/db-sdb? [y/n]: > y > [root@osd0 ~]# vgremove vg_db_tmp > Volume group "vg_db_tmp" successfully removed > [root@osd0 ~]# pvremove /dev/sdd > Labels on physical volume "/dev/sdd" successfully wiped. > ``` > add the new osd (should be done by re-running the playbook): > ``` > [root@osd0 ~]# ceph-volume lvm batch --bluestore --yes /dev/sdb /dev/sdc > --db-devices /dev/sda > --> passed data devices: 2 physical, 0 LVM > --> relative data size: 1.0 > --> passed block_db devices: 1 physical, 0 LVM > ... omitted output ... > --> ceph-volume lvm activate successful for osd ID: 1 > --> ceph-volume lvm create successful for: /dev/sdc > [root@osd0 ~]# > ``` > new lsblk output: > ``` > [root@osd0 ~]# lsblk > NAME >MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > sda > 8:00 50G 0 disk > |-ceph--8d085f45--939c--4a65--a577--d21fa146d7d6-osd--db--cd34400d--daf2--450f--97d9--d561e7a43d1a >252:00 25G 0 lvm > `-ceph--8d085f45--939c--4a65--a577--d21fa146d7d6-osd--db--bb30e5aa--a634--4c52--8b99--a222c03c18e3 >252:30 25G 0 lvm > sdb > 8:16 0 50G 0 disk > `-ceph--4c77295c--28a5--440a--9561--b9dc4c814e36-osd--block--70fd3b96--7bb2--4ae3--a0f8--4d18748186f9 > 252:10 50G 0 lvm > sdc > 8:32 0 50G 0 disk > `-ceph--5255bfbb--f133--4954--aaa8--35e2643ed491-osd--block--9e67ea46--2409--45f8--83e1--f66a42a6d9d0 > 252:2 0 50G 0 lvm > sdd > 8:48 0 50G 0 disk > vda > 253:00 11G 0 disk > `-vda1 >253:10 10G 0 part / > ``` > If you plan to re-run the playbook, do not forget to update your group_vars > to reflect the new topology: > ``` > # cat group_vars/all > --- > devices: > - /dev/sdb > - /dev/sdc > dedicated_devices: > - /dev/sda > ``` > You might want to use some osd flags (noout, etc..) in order to avoid > unnecessary data migration. > Regards, > On Tue, 17 Jan 2023 at 18:39, Len Kimms wrote: > > Hello all, > > > > we’ve set up a new Ceph cluster with a number of nodes which are all > > identically configured. > > There is one device vda which should act as WAL device for all other > > devices. Additionally, there are four other devices vdb, vdc, vdd, vde > > which use vda as WAL. > > The whole cluster was set up using ceph-ansible (branch stable-7.0) and > > Ceph version 17.2.0. > > Device configuration in osds.yml looks as follows: > >devices: [/dev/vdb, /dev/vdc, /dev/vdd, /dev/vde] > >bluestore_wal_devices: [/dev/vda] > > As expected vda contains four logical volumes for WAL each 1/4 of the > > overall vda disk size (‘ceph-ansible/group_vars/all.yml’ has default > > ‘block_db_size: -1’). > > > > After the initial setup, we’ve added an additional device vdf which should > > become a new OSD. The new OSD should use vda for WAL as well. This means > > the previous four WAL LVs have to be resized down to 1/5 and a new LV has > > to be added. > > > > Is it possible to retroactively add a new device to an already provisioned > > WAL device? > > > > We suspect that this is not possible because the ceph-bluestore-tool does > > not provide any way to shrink an existing BlueFS device. Only expanding is > > currently possible ( > > https://docs.ceph.com/en/quincy/man/8/ceph-bluestore-tool/). > > Simply adding the new device to the devices list and rerunning the > > playbook does nothing. And so does only setting “devices: [/dev/vdf]” and > > “bluestore_wal_devices: [/dev/vda]”. In both cases vda is rejected because > > “Insufficient space (<10 extents) on vgs” which makes sense because vda is > > already fully used by the previous four OSD WALs. > > > > Thanks for the help and kind regards. > > > > > > Additional notes: > > - We’re testing pre-production on an emulated cluster hence the device > > names vdx and unusually small device sizes. > > - The output of `lsblk` after the initial setup looks as follows: > > ``` > > vda > >252:008G 0 disk > > ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa8-osd--wal--3677c354--8d7d--4db9--a2b7--68aeb8248d40 > > 253:202G 0 lvm > > ├─ceph--36607c7f--e51c--452e--a44a--225d8d0b0aa
