Ah. I think the FIO tests on page 3 are intended to be run *before* deploying OSDs, and the rados bench tests on page 6 against the cluster after deployment.
When running rados bench, be sure to clean up afterwards. It’s all to easy to clutter up a RADOS pool with orphaned benchmark objects. > On Jul 23, 2025, at 9:24 AM, GLE, Vivien <vivien....@inist.fr> wrote: > > I followed this doc from Proxmox to test IO on my OSD > https://www.proxmox.com/images/download/pve/docs/Proxmox-VE-Ceph-Benchmark-202312-rev0.pdf > > Did it on 3 OSD to bench disk on a testing ceph cluster and one went down > > Wanted to see the difference between a rados bench and fio > > Vivien > De : Anthony D'Atri <a...@dreamsnake.net <mailto:a...@dreamsnake.net>> > Envoyé : mercredi 23 juillet 2025 14:06:55 > À : GLE, Vivien > Cc : Sinan Polat; ceph-users@ceph.io <mailto:ceph-users@ceph.io> > Objet : Re: [ceph-users] Ceph OSD down (unable to mount object store) > > You ran a write fio job on the underlying device. That would make any > software unhappy. > > Did you mean to run a read test? Or to test on an RBD volume or a filesystem > built within an RBD volume? > > > > > > > Thanks for your answer ! > > > > > > Do you know by any chance why fio does this ? Are fio and ceph incompatible > > ? > > > > > > Let's destroy this OSD then > > > > > > Vivien > > > > ________________________________ > > De : Sinan Polat <sinan86po...@gmail.com <mailto:sinan86po...@gmail.com>> > > Envoyé : mercredi 23 juillet 2025 13:57:07 > > À : GLE, Vivien > > Cc : ceph-users@ceph.io <mailto:ceph-users@ceph.io> > > Objet : Re: [ceph-users] Ceph OSD down (unable to mount object store) > > > > Hi Vivien, > > > > Your fio test has very likely destroyed the Ceph OSD block device and the > > problem is not just the symlink, it's data corruption on the underlying > > device. > > > > Zap the drive, recreate the OSD and let your cluster rebalance. > > > > Sinan > > > > Op wo 23 jul 2025 om 14:10 schreef GLE, Vivien <vivien....@inist.fr > > <mailto:vivien....@inist.fr><mailto:vivien....@inist.fr>>: > > Hi, > > > > > > I did a fio bench and believe that it destroy one of my OSD, this is the > > command used : > > > > > > fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write > > --bs=4k --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio > > fio --ioengine=libaio --filename=/dev/sda --direct=1 --sync=1 --rw=write > > --bs=4M --numjobs=1 --iodepth=1 --runtime=60 --time_based --name=fio > > > > > > Log file after the command : > > > > 2025-07-23T08:41:44.448+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 > > /var/lib/ceph/osd/ceph-2/block) close > > 2025-07-23T08:41:44.719+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 > > /var/lib/ceph/osd/ceph-2/block) open path /var/lib/ceph/osd/ceph-2/block > > 2025-07-23T08:41:44.719+0000 734bf7f6e680 0 bdev(0x59ba86d4ee00 > > /var/lib/ceph/osd/ceph-2/block) ioctl(F_SET_FILE_RW_HINT) on > > /var/lib/ceph/osd/ceph-2/block failed: (22) Invalid argument > > 2025-07-23T08:41:44.720+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 > > /var/lib/ceph/osd/ceph-2/block) open size 1000203091968 (0xe8e0c00000, 932 > > GiB) block_size 4096 (4 KiB) rotational device, discard supported > > 2025-07-23T08:41:44.723+0000 734bf7f6e680 -1 > > bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode > > label /var/lib/ceph/osd/ceph-2/block at offset 66: void > > bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) > > decode past end of struct encoding: Malformed input [buffer:3] > > 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 > > bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label unable to decode > > label /var/lib/ceph/osd/ceph-2/block at offset 4096: End of buffer > > [buffer:2] > > 2025-07-23T08:41:44.724+0000 734bf7f6e680 -1 > > bluestore(/var/lib/ceph/osd/ceph-2) _check_main_bdev_label not all labels > > read properly > > 2025-07-23T08:41:44.724+0000 734bf7f6e680 1 bdev(0x59ba86d4ee00 > > /var/lib/ceph/osd/ceph-2/block) close > > 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 osd.2 0 OSD:init: unable to > > mount object store > > 2025-07-23T08:41:44.983+0000 734bf7f6e680 -1 ** ERROR: osd init failed: > > (5) Input/output error > > > > > > > > After checking I saw that the block soft link might be wrong > > > > On a healthy osd : > > > > # ll /var/lib/ceph/cluster-id/osd.5/ > > total 72 > > drwx------ 2 167 167 4096 Jul 22 10:36 ./ > > drwx------ 12 167 167 4096 Jul 22 10:30 ../ > > lrwxrwxrwx 1 167 167 93 Jul 15 14:39 block -> > > /dev/ceph-c31f0e16-0460-4bc5-9470-468270b4c68a/osd-block-72aa3074-e2f9-45f8-a468-03c02d36f1de > > > > > > On my breaking osd : > > > > /var/lib/ceph/cluster-id/osd.2# ll > > total 72 > > drwx------ 2 167 167 4096 Jul 23 10:41 ./ > > drwx------ 11 167 167 4096 Jul 22 10:35 ../ > > lrwxrwxrwx 1 167 167 111 Jul 23 10:41 block -> > > /dev/mapper/ceph--bd0ca671--2b89--4530--be2b--f41443822a91-osd--block--342578c3--4603--4023--a564--8fca6dcc1040 > > > > Is there a correct way of changing it or I'm going in the wrong direction ? > > > > Thanks > > > > Vivien > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > <mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-le...@ceph.io > > <mailto:ceph-users-le...@ceph.io><mailto:ceph-users-le...@ceph.io> > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-le...@ceph.io > > <mailto:ceph-users-le...@ceph.io> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
