Dear Cephalopodians, we had our cluster (still in testing phase) configured for automatic updates so we got 12.2.3 "automagically" when it was released.
In /etc/sysconfig/ceph, we still have the default:
CEPH_AUTO_RESTART_ON_UPGRADE=no
so as expected, services were not restarted.
However, as soon as scrubs started to run, we got many scrub errors and
inconsistent PGs.
Looking into the logs, I found that some ceph-osd processes (still running as
12.2.2) tried to load
the compression library (libsnappy) dynamically, and refused to do so since it
was already updated to 12.2.3 on disk.
This appears to have caused the OSD to report read errors.
The situation was reasonably easy to fix (i.e. just restart all ceph-osd
processes, and re-run a deep scrub some of the inconsistent PGs).
Still, I wonder whether this could be prevented by loading the libraries at OSD
startup (and never unloading them),
or by shutting down the OSD in case of a library load failure.
Did anybody else experience this as of yet?
We will work around it either by version pinning or
CEPH_AUTO_RESTART_ON_UPGRADE=yes (not decided yet).
Cheers,
Oliver
smime.p7s
Description: S/MIME Cryptographic Signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
