Hi,
attachment stripped. Here is the log:
http://www-f9.ijs.si/~andrej/ceph-osd.611.log-20211220-short.gz
Andrej
On 12/20/21 09:17, Andrej Filipcic wrote:
Hi,
When upgrading to 16.2.7 from 16.2.6, 8 out of ~1600 OSDs failed to
start. The first 16.2.7 startup crashes here:
2021-12-19T09:52:34.128+0100 7ff7104c0080 1 bluefs mount
2021-12-19T09:52:34.129+0100 7ff7104c0080 1 bluefs _init_alloc
shared, id 1, capacity 0xe8d7fc00000, block size 0x10000
2021-12-19T09:52:34.238+0100 7ff7104c0080 1 bluefs mount
shared_bdev_used = 0
2021-12-19T09:52:34.238+0100 7ff7104c0080 1
bluestore(/var/lib/ceph/osd/ceph-611) _prepare_db_environment set
db_paths to db,15200851643596 db.slow,15200851643596
2021-12-19T09:52:34.257+0100 7ff7104c0080 -1 rocksdb: verify_sharding
unable to list column families: Corruption: CURRENT file does not end
with newline
2021-12-19T09:52:34.257+0100 7ff7104c0080 -1
bluestore(/var/lib/ceph/osd/ceph-611) _open_db erroring opening db:
2021-12-19T09:52:34.257+0100 7ff7104c0080 1 bluefs umount
I could export the rocksdb, and the contents of the CURRENT file is
corruped, I understand it should contain the MANIFEST-* info.
I have attached the full osd log of one failure, the others failed OSD
all fail for the same reason.
Any hint? for now, I keep those osds off if they can be further debugged.
(resending with shortened log)
Best regards,
Andrej
--
_____________________________________________________________
prof. dr. Andrej Filipcic, E-mail: andrej.filip...@ijs.si
Department of Experimental High Energy Physics - F9
Jozef Stefan Institute, Jamova 39, P.o.Box 3000
SI-1001 Ljubljana, Slovenia
Tel.: +386-1-477-3674 Fax: +386-1-477-3166
-------------------------------------------------------------
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
