Hi Parshant thanks for the update.
@Yuri Weinstein <ywein...@redhat.com> Is someone already looking into the cryptsetup issue I think it got introduced with this change here "ceph-volume: fix set_dmcrypt_no_workqueue() (pr#58997 <https://github.com/ceph/ceph/pull/58997>, Guillaume Abrioux)" Do you know if we have to wait for the next patch release or will there be a hotfix release soon? BR Stephan Am Di., 15. Apr. 2025 um 23:05 Uhr schrieb Prashant Dhange < pdha...@redhat.com>: > Hi Stephan, > > Yes, the limitation to IPv4-only subnet checks is understood and should be > addressed by the reef backport PR: https://github.com/ceph/ceph/pull/62814. > This PR includes the necessary changes to handle IPv6 subnets correctly. > > Regards, > Prashant > > > On Fri, Apr 11, 2025 at 12:36 PM Stephan Hohn <step...@gridscale.io> > wrote: > >> Ok the two issues I see with reef release v18.2.5 >> >> - Subnet check seems to be ipv4 only which leads to e.g "public address is >> not in 'fd01:1:f00f:443::/64' subnet" warnings on ipv6 only clusters. >> >> >> - >> >> common/pick_address: check if address in subnet all public address ( >> pr#57590 <https://github.com/ceph/ceph/pull/57590>, Nitzan Mordechai) >> - >> >> osd: Report health error if OSD public address is not within subnet ( >> pr#55697 <https://github.com/ceph/ceph/pull/55697>, Prashant D) >> >> - cryptsetup version check isn't working at least in the container image >> of >> v18.2.5 ( >> >> https://github.com/ceph/ceph/blob/reef/src/ceph-volume/ceph_volume/util/encryption.py >> ) >> which leads to encrypted osds not starting due to "'Error while checking >> cryptsetup version.\n', '`cryptsetup --version` output:\n', 'cryptsetup >> 2.7.2 flags: UDEV BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY '" >> >> Happy to help with logs etc. >> >> BR >> >> Stephan >> >> >> >> Am Fr., 11. Apr. 2025 um 09:11 Uhr schrieb Stephan Hohn < >> step...@gridscale.io>: >> >> > Hi all, >> > >> > started an update on our staging cluster from v18.2.4 --> v18.2.5 >> > >> > ~# ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.5 Mons and >> Mgr >> > went fine but osds not coming up with v18.2.5 Apr 11 06:59:56 >> > 0cc47a6df14e podman[263290]: 2025-04-11 06:59:56.697993041 +0000 UTC >> > m=+0.057869056 image pull quay.io/ceph/ceph:v18.2.5 >> > Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 >> 06:59:56.778833855 >> > +0000 UTC m=+0.138709869 container init >> > 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, CEPH_GIT_REPO= >> > https://github.com/ceph/ceph.git, OSD_FLAVOR=default, >> > org.label-schema.schema-version=1.0, GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > org.opencontainers.image.documentation=https://docs.ceph.com/, >> > CEPH_REF=reef, org.label-schema.vendor=CentOS, ceph=True, >> > org.label-schema.name=CentOS Stream 9 Base Image, >> > io.buildah.version=1.39.3, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= >> > quay.io/centos/centos:stream9, org.opencontainers.image.authors=Ceph >> > Release Team <ceph-maintain...@ceph.io>, >> org.label-schema.license=GPLv2, >> > org.label-schema.build-date=20250325) >> > Apr 11 06:59:56 0cc47a6df14e podman[263290]: 2025-04-11 >> 06:59:56.790749299 >> > +0000 UTC m=+0.150625308 container start >> > 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, >> > org.label-schema.build-date=20250325, >> org.opencontainers.image.authors=Ceph >> > Release Team <ceph-maintain...@ceph.io>, >> org.label-schema.license=GPLv2, >> > org.label-schema.schema-version=1.0, ceph=True, CEPH_REF=reef, >> > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, OSD_FLAVOR=default, >> > org.label-schema.name=CentOS Stream 9 Base Image, >> > org.opencontainers.image.documentation=https://docs.ceph.com/, >> > GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > io.buildah.version=1.39.3, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= >> > quay.io/centos/centos:stream9, org.label-schema.vendor=CentOS) >> > Apr 11 06:59:56 0cc47a6df14e bash[263290]: >> > 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 >> > Apr 11 06:59:56 0cc47a6df14e >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: >> > 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 >> > bluestore(/var/lib/ceph/osd/ceph-20/block) _read_bdev_label failed to >> open >> > /var/lib/ceph/osd/ceph-20/block: (2) No such file or directory >> > Apr 11 06:59:56 0cc47a6df14e >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20[263380]: >> > 2025-04-11T06:59:56.817+0000 7b72d0abc740 -1 ** ERROR: unable to open >> OSD >> > superblock on /var/lib/ceph/osd/ceph-20: (2) No such file or directory >> > Apr 11 06:59:56 0cc47a6df14e systemd[1]: Started Ceph osd.20 for >> > 03977a23-f00f-4bb0-b9a7-de57f40ba853. >> > Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 >> 06:59:56.90105365 >> > +0000 UTC m=+0.076310419 container died >> > 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20) >> > Apr 11 06:59:56 0cc47a6df14e podman[263399]: 2025-04-11 >> 06:59:56.948423169 >> > +0000 UTC m=+0.123679914 container remove >> > 5db97f7e32705cc0e8fee1bc5741dfbd97ffa430b8fb5a1cfe19b768aed78b23 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20, ceph=True, >> > io.buildah.version=1.39.3, org.label-schema.name=CentOS Stream 9 Base >> > Image, org.opencontainers.image.authors=Ceph Release Team < >> > ceph-maintain...@ceph.io>, CEPH_REF=reef, GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, FROM_IMAGE= >> > quay.io/centos/centos:stream9, OSD_FLAVOR=default, >> > org.label-schema.build-date=20250325, org.label-schema.vendor=CentOS, >> > org.label-schema.schema-version=1.0, >> org.opencontainers.image.documentation= >> > https://docs.ceph.com/, org.label-schema.license=GPLv2, CEPH_GIT_REPO= >> > https://github.com/ceph/ceph.git) >> > Apr 11 06:59:56 0cc47a6df14e systemd[1]: >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Main process >> > exited, code=exited, status=1/FAILURE >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:57.495704469 >> > +0000 UTC m=+0.105177519 container create >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, >> > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, >> GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 >> > Base Image, FROM_IMAGE=quay.io/centos/centos:stream9, >> > io.buildah.version=1.39.3, org.opencontainers.image.documentation= >> > https://docs.ceph.com/, org.label-schema.schema-version=1.0, >> > org.label-schema.build-date=20250325, ceph=True, CEPH_REF=reef, >> > org.label-schema.license=GPLv2, org.opencontainers.image.authors=Ceph >> > Release Team <ceph-maintain...@ceph.io>, OSD_FLAVOR=default) >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:57.447692243 >> > +0000 UTC m=+0.057165313 image pull quay.io/ceph/ceph:v18.2.5 >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:57.532516757 >> > +0000 UTC m=+0.141989804 container init >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, >> > CEPH_REF=reef, org.label-schema.license=GPLv2, >> > org.label-schema.schema-version=1.0, CEPH_GIT_REPO= >> > https://github.com/ceph/ceph.git, GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > org.label-schema.vendor=CentOS, OSD_FLAVOR=default, >> > org.opencontainers.image.authors=Ceph Release Team < >> > ceph-maintain...@ceph.io>, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, >> org.label-schema.name=CentOS >> > Stream 9 Base Image, org.opencontainers.image.documentation= >> > https://docs.ceph.com/, ceph=True, io.buildah.version=1.39.3, >> > org.label-schema.build-date=20250325, FROM_IMAGE= >> > quay.io/centos/centos:stream9) >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:57.542542891 >> > +0000 UTC m=+0.152015938 container start >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, >> > GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > org.label-schema.build-date=20250325, OSD_FLAVOR=default, CEPH_REF=reef, >> > ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, >> > org.opencontainers.image.authors=Ceph Release Team < >> > ceph-maintain...@ceph.io>, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, >> > org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, >> > org.opencontainers.image.documentation=https://docs.ceph.com/, >> FROM_IMAGE= >> > quay.io/centos/centos:stream9, org.label-schema.schema-version=1.0, >> > org.label-schema.license=GPLv2, CEPH_GIT_REPO= >> > https://github.com/ceph/ceph.git) >> > Apr 11 06:59:57 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:57.542633919 >> > +0000 UTC m=+0.152106981 container attach >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, >> > FROM_IMAGE=quay.io/centos/centos:stream9, >> > org.label-schema.schema-version=1.0, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, OSD_FLAVOR=default, >> > org.opencontainers.image.documentation=https://docs.ceph.com/, >> > org.label-schema.vendor=CentOS, io.buildah.version=1.39.3, ceph=True, >> > org.label-schema.license=GPLv2, org.label-schema.build-date=20250325, >> > org.opencontainers.image.authors=Ceph Release Team < >> > ceph-maintain...@ceph.io>, GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > CEPH_GIT_REPO=https://github.com/ceph/ceph.git, CEPH_REF=reef, >> > org.label-schema.name=CentOS Stream 9 Base Image) >> > Apr 11 06:59:58 0cc47a6df14e >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: >> > Running command: /usr/sbin/cryptsetup remove >> > UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x >> > Apr 11 06:59:58 0cc47a6df14e bash[263966]: Running command: >> > /usr/sbin/cryptsetup remove UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x >> > Apr 11 06:59:58 0cc47a6df14e >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: >> > stderr: Device UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. >> > Apr 11 06:59:58 0cc47a6df14e bash[263966]: stderr: Device >> > UL0ODv-5ESt-FfPX-aGAK-0lfX-hRF5-14K71x is not active. >> > Apr 11 06:59:58 0cc47a6df14e >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate[264056]: --> >> > command returned non-zero exit status: 4 >> > Apr 11 06:59:58 0cc47a6df14e bash[263966]: --> command returned non-zero >> > exit status: 4 >> > Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:58.401862967 >> > +0000 UTC m=+1.011336030 container died >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate) >> > Apr 11 06:59:58 0cc47a6df14e podman[263966]: 2025-04-11 >> 06:59:58.456490831 >> > +0000 UTC m=+1.065963881 container remove >> > d96a2746c9b6ac37f42e1beaac9f572d22558c16d662dfaff994d1d90c611ad8 (image= >> > quay.io/ceph/ceph:v18.2.5, >> > name=ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853-osd-20-deactivate, >> > org.opencontainers.image.authors=Ceph Release Team < >> > ceph-maintain...@ceph.io>, CEPH_GIT_REPO= >> https://github.com/ceph/ceph.git, >> > org.label-schema.build-date=20250325, GANESHA_REPO_BASEURL= >> > >> https://buildlogs.centos.org/centos/$releasever-stream/storage/$basearch/nfsganesha-5/ >> , >> > io.buildah.version=1.39.3, org.label-schema.schema-version=1.0, >> ceph=True, >> > org.label-schema.vendor=CentOS, org.opencontainers.image.documentation= >> > https://docs.ceph.com/, OSD_FLAVOR=default, CEPH_REF=reef, >> > org.label-schema.license=GPLv2, >> > CEPH_SHA1=a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1, >> org.label-schema.name=CentOS >> > Stream 9 Base Image, FROM_IMAGE=quay.io/centos/centos:stream9) >> > Apr 11 06:59:58 0cc47a6df14e systemd[1]: >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Failed with >> > result 'exit-code'. >> > Apr 11 06:59:58 0cc47a6df14e systemd[1]: >> > ceph-03977a23-f00f-4bb0-b9a7-de57f40ba853@osd.20.service: Consumed >> 2.252s >> > CPU time. >> > >> > >> > I was able to bring osds up again by redewploying with the old container >> > image v18.2.4 >> > >> > ~# ceph orch daemon redeploy osd.16 --image quay.io/ceph/ceph:v18.2.4 >> > >> > Everything is up again but seeing the this error now. ~# ceph health >> > detail HEALTH_ERR noout flag(s) set; 24 osds(s) are not reachable [WRN] >> > OSDMAP_FLAGS: noout flag(s) set [ERR] OSD_UNREACHABLE: 24 osds(s) are >> not >> > reachable osd.0's public address is not in 'fd01:1:f00f:443::/64' subnet >> > osd.1's public address is not in 'fd01:1:f00f:443::/64' subnet >> > ... >> > >> > ~# ceph config dump | grep netw global advanced public_network >> > fd01:1:f00f:443::/64 * mon advanced public_network fd01:1:f00f:443::/64 >> * >> > ~# ss -tulpn |grep osd tcp LISTEN 0 512 [fd01:1:f00f:443::10]:6828 >> [::]:* >> > users:(("ceph-osd",pid=204834,fd=22)) tcp LISTEN 0 512 >> > [fd01:1:f00f:443::10]:6829 [::]:* users:(("ceph-osd",pid=204834,fd=23)) >> > >> > Looks like some see the same problem with latest squid release v19.2.2 >> as >> > well. >> > BR Stephan >> > >> > >> > >> > >> > >> > Am Do., 10. Apr. 2025 um 22:45 Uhr schrieb Yuri Weinstein < >> > ywein...@redhat.com>: >> > >> >> We're happy to announce the 5th point release in the Reef series. >> >> >> >> We recommend users to update to this release. >> >> For detailed release notes with links & changelog please refer to the >> >> official blog entry at >> >> https://ceph.io/en/news/blog/2025/v18-2-5-reef-released/ >> >> >> >> >> >> Notable Changes >> >> --------------- >> >> * RBD: The ``try-netlink`` mapping option for rbd-nbd has become the >> >> default >> >> and is now deprecated. If the NBD netlink interface is not supported >> by >> >> the >> >> kernel, then the mapping is retried using the legacy ioctl interface. >> >> >> >> * RADOS: A new command, `ceph osd rm-pg-upmap-primary-all`, has been >> >> added that allows >> >> users to clear all pg-upmap-primary mappings in the osdmap when >> desired. >> >> >> >> Related trackers: >> >> - https://tracker.ceph.com/issues/67179 >> >> - https://tracker.ceph.com/issues/66867 >> >> >> >> Getting Ceph >> >> ------------ >> >> * Git at git://github.com/ceph/ceph.git >> >> * Tarball at >> https://download.ceph.com/tarballs/ceph_18.2.5.orig.tar.gz >> >> * Containers at https://quay.io/repository/ceph/ceph >> >> * For packages, see >> https://docs.ceph.com/en/latest/install/get-packages/ >> >> * Release git sha1: a5b0e13f9c96f3b45f596a95ad098f51ca0ccce1 >> >> _______________________________________________ >> >> Ceph-announce mailing list -- ceph-annou...@ceph.io >> >> To unsubscribe send an email to ceph-announce-le...@ceph.io >> >> >> > >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-le...@ceph.io >> >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
