Hi,
can you show the output of 'ceph orch ls osd --export'? I would look
in the cephadm.log and ceph-volume.log on that node as well as in the
active mgr log. If you already have an osd service that would pick up
this osd, and you zapped it, its creation might have been interrupted.
If you create all your OSDs manually with 'ceph orch daemon add
osd...' this theory doesn't make much sense.
The replaced OSD would not necessarily get the same ID as before (2),
just to mention it.
Zitat von Gustavo Garcia Rondina <grond...@uchicago.edu>:
Alright, now `ceph orch device ls` show the disk as locked:
HOST PATH TYPE DEVICE ID
SIZE AVAILABLE REFRESHED REJECT REASONS
node-osd1 /dev/sdah hdd
LENOVO_ST18000NM004J_ZR5F5TH00000W413B814 18.0T No 10m ago
locked
I also noticed that in `ceph osd crush dump` where osd.2 was listed
before in the device list now there's a mysterious "device2":
{
"devices": [
{
"id": 0,
"name": "osd.0",
"class": "hdd"
},
{
"id": 1,
"name": "osd.1",
"class": "hdd"
},
{
"id": 2,
"name": "device2"
},
{
"id": 3,
"name": "osd.3",
"class": "hdd"
},
...
Even though lsblk shows a Ceph LVM volume created on the disk, it
seems that it was not complete:
# lsblk /dev/sdah
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdah
66:16 0 16.4T 0 disk
└─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0
lvm
# blkid /dev/sdah
# lvs | grep 157c7c44
# vgs | grep 157c7c44
# pvs | grep sdah
#
It seems that OSD creation was in progress and somehow only got
halfway there.
Could it have to do with the fact that our cluster is not in the
best shape, though it is recovery well? ceph -s below:
cluster:
id: 26315dca-383a-11ee-9d49-00620b4c2392
health: HEALTH_ERR
987 scrub errors
Possible data damage: 17 pgs inconsistent
2192 pgs not deep-scrubbed in time
2192 pgs not scrubbed in time
services:
mon: 5 daemons, quorum
node-admin1,node-admin2,node-osd1,node-osd2,node-osd3 (age 17M)
mgr: node-admin2.sipadf(active, since 17M), standbys: node-admin1.nwaovh
mds: 2/2 daemons up, 2 standby
osd: 167 osds: 167 up (since 3d), 167 in (since 2h); 237 remapped pgs
data:
volumes: 2/2 healthy
pools: 9 pools, 2273 pgs
objects: 475.75M objects, 1.1 PiB
usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail
pgs: 73449562/2835243480 objects misplaced (2.591%)
2024 active+clean
228 active+remapped+backfilling
12 active+clean+inconsistent
5 active+remapped+inconsistent+backfilling
4 active+remapped+backfill_wait
io:
client: 56 MiB/s wr, 0 op/s rd, 97 op/s wr
recovery: 1.0 GiB/s, 440 objects/s
progress:
Global Recovery Event (7w)
[=========================...] (remaining: 6d)
________________________________
From: Gustavo Garcia Rondina <grond...@uchicago.edu>
Sent: Thursday, March 6, 2025 4:04 PM
To: ceph-users@ceph.io <ceph-users@ceph.io>
Subject: [ceph-users] Created no osd(s) on host, already created?
Hello list,
We have a Ceph cluster with two management nodes and six data nodes.
Each data node has 28 HDD disks. One disk recently failed in one of
the nodes, corresponding to osd.2. To replace the disk, we took the
osd.2 out, stopped it, and after a few days removed it, basically:
ceph osd out osd.2
ceph osd ok-to-stop osd.2
Once Ok to stop, then:
ceph orch daemon stop osd.2
Once stopped:
ceph osd crush remove osd.2
ceph auth del osd.2
ceph osd rm osd.2
Then checked `ceph osd tree`, `ceph orch ps`, `ceph -s`, and all
confirmed that the OSD was gone. We then proceeded to physically
replace the failed drive on the node. It was /dev/sdac before, and
after replacing it (hot swap) the system identified it as /dev/sdah.
I zapped it on the node with `sgdisk --zap-all /dev/sdah` and, back
on the management node, I could see that the disk was now showing up
and marked as available with `ceph orch device ls`.
Then I proceeded to add a new OSD to that disk with: `ceph orch
daemon add osd node-osd1:/dev/sdah`, which then failed with:
Created no osd(s) on host server-osd1; already created?
Which was rather strange, because total OSDs was still showing as 1
less, no new OSD was showing up in `ceph osd tree`. What was even
odder is that the disk that was showing as available in `ceph orch
device ls` now shows as *not* available, and looking at lsblk's
output in the node it seems that it was populated by Ceph:
# lsblk /dev/sdah
NAME
MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sdah
66:16 0 16.4T 0 disk
└─ceph--157c7c44--b519--4f6d--a54b--ecd466cf81d0-osd--block--10ada574--b99c--466d--8ad6--2c97d17d1f66 253:59 0 16.4T 0
lvm
Any hints on how to proceed to get the OSD added back with this disk?
Thank you for any suggestions!
-
Gustavo
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
