I haven’t attempted the remaining upgrade just yet. I wanted to check on this before proceeding. Things seem “stable” in the sense that I’m running VMs and all volumes and images are still functioning. I’m using whatever would have been the default from 16.2.14. It seems to be from time to time because I receive nagios alerts, which seem to eventually clear and then reappear.
HEALTH_WARN Failed to apply 1 service(s): osd.cost_capacity
[WRN] CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.cost_capacity
osd.cost_capacity: cephadm exited with an error code: 1, stderr:Inferring
config /var/lib/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1/mon.cn02/config
Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:47de8754d1f72fadb61523247c897fdf673f9a9689503c64ca8384472d232c5c
-e NODE_NAME=cn02.ceph.xyz.corp -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity
-e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1:/var/run/ceph:z -v
/var/log/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1:/var/log/ceph:z -v
/var/lib/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmp49jj8zoh:/etc/ceph/ceph.conf:z -v
/tmp/ceph-tmp_9k8v5uj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
quay.io/ceph/ceph@sha256:47de8754d1f72fadb61523247c897fdf673f9a9689503c64ca8384472d232c5c
lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf --dmcrypt
--yes --no-systemd
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 33, in <module>
/usr/bin/podman: stderr sys.exit(load_entry_point('ceph-volume==1.0.0',
'console_scripts', 'ceph-volume')())
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 54, in __init__
/usr/bin/podman: stderr self.main(self.argv)
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/decorators.py", line 59, in
newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/main.py", line 166, in main
/usr/bin/podman: stderr terminal.dispatch(self.mapper, subcommand_args)
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 194, in
dispatch
/usr/bin/podman: stderr instance.main()
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/main.py", line 46, in
main
/usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv)
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/terminal.py", line 192, in
dispatch
/usr/bin/podman: stderr instance = mapper.get(arg)(argv[count:])
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/devices/lvm/batch.py", line 325,
in __init__
/usr/bin/podman: stderr self.args = parser.parse_args(argv)
/usr/bin/podman: stderr File "/usr/lib64/python3.9/argparse.py", line 1825, in
parse_args
/usr/bin/podman: stderr args, argv = self.parse_known_args(args, namespace)
/usr/bin/podman: stderr File "/usr/lib64/python3.9/argparse.py", line 1858, in
parse_known_args
/usr/bin/podman: stderr namespace, args = self._parse_known_args(args,
namespace)
/usr/bin/podman: stderr File "/usr/lib64/python3.9/argparse.py", line 2067, in
_parse_known_args
/usr/bin/podman: stderr start_index = consume_optional(start_index)
/usr/bin/podman: stderr File "/usr/lib64/python3.9/argparse.py", line 2007, in
consume_optional
/usr/bin/podman: stderr take_action(action, args, option_string)
/usr/bin/podman: stderr File "/usr/lib64/python3.9/argparse.py", line 1935, in
take_action
/usr/bin/podman: stderr action(self, namespace, argument_values, option_string)
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/util/arg_validators.py", line 17,
in __call__
/usr/bin/podman: stderr set_dmcrypt_no_workqueue()
/usr/bin/podman: stderr File
"/usr/lib/python3.9/site-packages/ceph_volume/util/encryption.py", line 54, in
set_dmcrypt_no_workqueue
/usr/bin/podman: stderr raise RuntimeError('Error while checking cryptsetup
version.\n',
/usr/bin/podman: stderr RuntimeError: ('Error while checking cryptsetup
version.\n', '`cryptsetup --version` output:\n', 'cryptsetup 2.7.2 flags: UDEV
BLKID KEYRING FIPS KERNEL_CAPI PWQUALITY ')
Traceback (most recent call last):
File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 11009, in <module>
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 10997, in main
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 2593, in _infer_config
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 2509, in _infer_fsid
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 2621, in _infer_image
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 2496, in _validate_fsid
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 7226, in
command_ceph_volume
File "/tmp/tmpedb1_faj.cephadm.build/__main__.py", line 2284, in call_throws
RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host
--stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume
--privileged --group-add=disk --init -e
CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:47de8754d1f72fadb61523247c897fdf673f9a9689503c64ca8384472d232c5c
-e NODE_NAME=cn02.ceph.xyz.corp -e CEPH_VOLUME_OSDSPEC_AFFINITY=cost_capacity
-e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v
/var/run/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1:/var/run/ceph:z -v
/var/log/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1:/var/log/ceph:z -v
/var/lib/ceph/95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1/crash:/var/lib/ceph/crash:z
-v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v
/run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v
/run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v
/tmp/ceph-tmp49jj8zoh:/etc/ceph/ceph.conf:z -v
/tmp/ceph-tmp_9k8v5uj:/var/lib/ceph/bootstrap-osd/ceph.keyring:z
quay.io/ceph/ceph@sha256:47de8754d1f72fadb61523247c897fdf673f9a9689503c64ca8384472d232c5c
lvm batch --no-auto /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf --dmcrypt
--yes --no-systemd
—
ceph orch ls osd --export
service_type: osd
service_id: all-available-devices
service_name: osd.all-available-devices
placement:
host_pattern: '*'
spec:
data_devices:
all: true
filter_logic: AND
objectstore: bluestore
---
service_type: osd
service_id: cost_capacity
service_name: osd.cost_capacity
placement:
host_pattern: '*'
spec:
data_devices:
rotational: 1
encrypted: true
filter_logic: AND
objectstore: bluestore
Thank you
-jeremy
> On Sunday, Apr 13, 2025 at 11:48 PM, Eugen Block <ebl...@nde.ag
> (mailto:ebl...@nde.ag)> wrote:
> Are you using Rook? Usually, I see this warning when a host is not
> reachable, for example during a reboot. But it also clears when the
> host comes back. Do you see this permanently or from time to time? It
> might have to do with the different Ceph versions, I'm not sure. But
> it shouldn't be a show stopper for the remaining upgrade. Or are you
> trying to deploy OSDs but it fails? You can paste
>
> ceph health detail
> ceph orch ls osd --export
>
> You can also scan the cephadm.log for any hints.
>
>
> Zitat von Jeremy Hansen <jer...@skidrow.la>:
>
> > This looks relevant.
> >
> > https://github.com/rook/rook/issues/13600#issuecomment-1905860331
> >
> > > On Sunday, Apr 13, 2025 at 10:08 AM, Jeremy Hansen
> > > <jer...@skidrow.la (mailto:jer...@skidrow.la)> wrote:
> > > I’m now seeing this:
> > >
> > > cluster:
> > > id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1
> > > health: HEALTH_WARN
> > > Failed to apply 1 service(s): osd.cost_capacity
> > >
> > >
> > > I’m assuming this is due to the fact that I’ve only upgraded mgr
> > > but I wanted to double check before proceeding with the rest of the
> > > components.
> > >
> > > Thanks
> > > -jeremy
> > >
> > >
> > >
> > >
> > > > On Sunday, Apr 13, 2025 at 12:59 AM, Jeremy Hansen
> > > <jer...@skidrow.la (mailto:jer...@skidrow.la)> wrote:
> > > > Updating mgr’s to 18.2.5 seemed to work just fine. I will go for
> > > the remaining services after the weekend. Thanks.
> > > >
> > > > -jeremy
> > > >
> > > >
> > > >
> > > > > On Thursday, Apr 10, 2025 at 6:37 AM, Eugen Block
> > > <ebl...@nde.ag (mailto:ebl...@nde.ag)> wrote:
> > > > > Glad I could help! I'm also waiting for 18.2.5 to upgrade our own
> > > > > cluster from Pacific after getting rid of our cache tier. :-D
> > > > >
> > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > > >
> > > > > > This seems to have worked to get the orch back up and put me back to
> > > > > > 16.2.15. Thank you. Debating on waiting for 18.2.5 to move forward.
> > > > > >
> > > > > > -jeremy
> > > > > >
> > > > > > > On Monday, Apr 07, 2025 at 1:26 AM, Eugen Block <ebl...@nde.ag
> > > > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > > > Still no, just edit the unit.run file for the MGRs to use a
> > > different
> > > > > > > image. See Frédéric's instructions (now that I'm re-reading it,
> > > > > > > there's a little mistake with dots and hyphens):
> > > > > > >
> > > > > > > # Backup the unit.run file
> > > > > > > $ cp /var/lib/ceph/$(ceph fsid)/mgr.ceph01.eydqvm/unit.run{,.bak}
> > > > > > >
> > > > > > > # Change container image's signature. You can get the
> > > signature of the
> > > > > > > version you
> > > > > > > want to reach from
> > > https://quay.io/repository/ceph/ceph?tab=tags. It's
> > > > > > > in the URL of a
> > > > > > > version.
> > > > > > > $ sed -i
> > > > > > >
> > > 's/ceph@sha256:e40c19cd70e047d14d70f5ec3cf501da081395a670cd59ca881ff56119660c8f/ceph@sha256:d26c11e20773704382946e34f0d3d2c0b8bb0b7b37d9017faa9dc11a0196c7d9/g'
> > > > > > > /var/lib/ceph/$(ceph fsid)/mgr.ceph01.eydqvm/unit.run
> > > > > > >
> > > > > > > # Restart the container (systemctl daemon-reload not needed)
> > > > > > > $ systemctl restart ceph-$(ceph fsid)(a)mgr.ceph01.eydqvm.service
> > > > > > >
> > > > > > > # Run this command a few times and it should show the new version
> > > > > > > ceph orch ps --refresh --hostname ceph01 | grep mgr
> > > > > > >
> > > > > > > To get the image signature, you can also look into the
> > > other unit.run
> > > > > > > files, a version tag would also work.
> > > > > > >
> > > > > > > It depends on how often you need the orchestrator to maintain the
> > > > > > > cluster. If you have the time, you could wait a bit longer for
> > > > > > > other
> > > > > > > responses. If you need the orchestrator in the meantime,
> > > you can roll
> > > > > > > back the MGRs.
> > > > > > >
> > > > > > >
> > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAIZ7IDCNI25KVYFCCCF6RJG/
> > > > > > >
> > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > > > > >
> > > > > > > > Thank you. The only thing I’m unclear on is the rollback
> > > to pacific.
> > > > > > > >
> > > > > > > > Are you referring to
> > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-manager-daemon
> > > > > > > >
> > > > > > > > Thank you. I appreciate all the help. Should I wait for Adam to
> > > > > > > > comment? At the moment, the cluster is functioning enough to
> > > > > > > > maintain running vms, so if it’s wise to wait, I can do that.
> > > > > > > >
> > > > > > > > -jeremy
> > > > > > > >
> > > > > > > > > On Monday, Apr 07, 2025 at 12:23 AM, Eugen Block
> > > > > > > > > <ebl...@nde.ag
> > > > > > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > > > > > I haven't tried it this way yet, and I had hoped that
> > > Adam would chime
> > > > > > > > > in, but my approach would be to remove this key (it's
> > > not present when
> > > > > > > > > no upgrade is in progress):
> > > > > > > > >
> > > > > > > > > ceph config-key rm mgr/cephadm/upgrade_state
> > > > > > > > >
> > > > > > > > > Then rollback the two newer MGRs to Pacific as
> > > described before. If
> > > > > > > > > they come up healthy, test if the orchestrator works
> > > properly first.
> > > > > > > > > For example, remove a node-exporter or crash or anything else
> > > > > > > > > uncritical and let it redeploy.
> > > > > > > > > If that works, try a staggered upgrade, starting with
> > > the MGRs only:
> > > > > > > > >
> > > > > > > > > ceph orch upgrade start --image <image-name> --daemon-types
> > > > > > > > > mgr
> > > > > > > > >
> > > > > > > > > Since there's no need to go to Quincy, I suggest to
> > > upgrade to Reef
> > > > > > > > > 18.2.4 (or you wait until 18.2.5 is released, which
> > > should be very
> > > > > > > > > soon), so set the respective <image-name> in the above
> > > > > > > > > command.
> > > > > > > > >
> > > > > > > > > If all three MGRs successfully upgrade, you can
> > > continue with the
> > > > > > > > > MONs, or with the entire rest.
> > > > > > > > >
> > > > > > > > > In production clusters, I usually do staggered
> > > upgrades, e. g. I limit
> > > > > > > > > the number of OSD daemons first just to see if they
> > > come up healthy,
> > > > > > > > > then I let it upgrade all other OSDs automatically.
> > > > > > > > >
> > > > > > > > >
> > > https://docs.ceph.com/en/latest/cephadm/upgrade/#staggered-upgrade
> > > > > > > > >
> > > > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > > > > > > >
> > > > > > > > > > Snipped some of the irrelevant logs to keep message size
> > > > > > > > > > down.
> > > > > > > > > >
> > > > > > > > > > ceph config-key get mgr/cephadm/upgrade_state
> > > > > > > > > >
> > > > > > > > > > {"target_name": "quay.io/ceph/ceph:v17.2.0", "progress_id":
> > > > > > > > > > "e7e1a809-558d-43a7-842a-c6229fdc57af", "target_id":
> > > > > > > > > >
> > > "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9",
> > > > > > > > > > "target_digests":
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > ["quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a",
> > >
> > > "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667"],
> > > "target_version": "17.2.0", "fs_original_max_mds": null,
> > > "fs_original_allow_standby_replay": null, "error": null, "paused": false,
> > > "daemon_types": null, "hosts": null, "services": null,
> > > "total_count":
> > > > > > > null,
> > > > > > > > > "remaining_count":
> > > > > > > > > > null}
> > > > > > > > > >
> > > > > > > > > > What should I do next?
> > > > > > > > > >
> > > > > > > > > > Thank you!
> > > > > > > > > > -jeremy
> > > > > > > > > >
> > > > > > > > > > > On Sunday, Apr 06, 2025 at 1:38 AM, Eugen Block
> > > <ebl...@nde.ag
> > > > > > > > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > > > > > > > Can you check if you have this config-key?
> > > > > > > > > > >
> > > > > > > > > > > ceph config-key get mgr/cephadm/upgrade_state
> > > > > > > > > > >
> > > > > > > > > > > If you reset the MGRs, it might be necessary to
> > > clear this key,
> > > > > > > > > > > otherwise you might end up in some inconsistency.
> > > Just to be sure.
> > > > > > > > > > >
> > > > > > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks. I’m trying to be extra careful since this
> > > cluster is
> > > > > > > > > > > > actually in use. I’ll wait for your feedback.
> > > > > > > > > > > >
> > > > > > > > > > > > -jeremy
> > > > > > > > > > > >
> > > > > > > > > > > > > On Saturday, Apr 05, 2025 at 3:39 PM, Eugen
> > > Block <ebl...@nde.ag
> > > > > > > > > > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > > > > > > > > > No, that's not necessary, just edit the
> > > unit.run file for
> > > > > > > > > the MGRs to
> > > > > > > > > > > > > use a different image. See Frédéric's instructions:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/32APKOXKRAIZ7IDCNI25KVYFCCCF6RJG/
> > > > > > > > > > > > >
> > > > > > > > > > > > > But I'm not entirely sure if you need to clear some
> > > > > > > > > config-keys first
> > > > > > > > > > > > > in order to reset the upgrade state. If I have
> > > time, I'll
> > > > > > > > > try to check
> > > > > > > > > > > > > tomorrow, or on Monday.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Zitat von Jeremy Hansen <jer...@skidrow.la>:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Would I follow this process to downgrade?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#manually-deploying-a-manager-daemon
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Saturday, Apr 05, 2025 at 2:04 PM, Jeremy
> > > > > > > > > > > > > > > Hansen
> > > > > > > > > > > > > > > <jer...@skidrow.la
> > > (mailto:jer...@skidrow.la)> wrote:
> > > > > > > > > > > > > > > ceph -s claims things are healthy:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > ceph -s
> > > > > > > > > > > > > > > cluster:
> > > > > > > > > > > > > > > id: 95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1
> > > > > > > > > > > > > > > health: HEALTH_OK
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > services:
> > > > > > > > > > > > > > > mon: 3 daemons, quorum cn01,cn03,cn02 (age 20h)
> > > > > > > > > > > > > > > mgr: cn03.negzvb(active, since 26m),
> > > standbys: cn01.tjmtph,
> > > > > > > > > > > > > > > cn02.ceph.xyz.corp.ggixgj
> > > > > > > > > > > > > > > mds: 1/1 daemons up, 2 standby
> > > > > > > > > > > > > > > osd: 15 osds: 15 up (since 19h), 15 in (since 14M)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > data:
> > > > > > > > > > > > > > > volumes: 1/1 healthy
> > > > > > > > > > > > > > > pools: 6 pools, 610 pgs
> > > > > > > > > > > > > > > objects: 284.59k objects, 1.1 TiB
> > > > > > > > > > > > > > > usage: 3.3 TiB used, 106 TiB / 109 TiB avail
> > > > > > > > > > > > > > > pgs: 610 active+clean
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > io:
> > > > > > > > > > > > > > > client: 255 B/s rd, 1.2 MiB/s wr, 10 op/s
> > > rd, 16 op/s wr
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > —
> > > > > > > > > > > > > > > How do I downgrade if the orch is down?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thank you
> > > > > > > > > > > > > > > -jeremy
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Saturday, Apr 05, 2025 at 1:56 PM, Eugen
> > > > > > > > > > > > > > > > Block
> > > > > > > > > <ebl...@nde.ag
> > > > > > > > > > > > > > > (mailto:ebl...@nde.ag)> wrote:
> > > > > > > > > > > > > > > > It would help if you only pasted the
> > > relevant parts.
> > > > > > > > > > > Anyway, these two
> > > > > > > > > > > > > > > > sections stand out:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > ---snip---
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:48.909+0000 7f26f0200700 > > > > > > > > > > > > > > > > 0
> > > > > > > > > > > [balancer INFO root]
> > > > > > > > > > > > > > > > Some PGs (1.000000) are unknown; try again later
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:48.917+0000
> > > 7f2663400700 -1 mgr
> > > > > > > > > > > load Failed to
> > > > > > > > > > > > > > > > construct class in 'cephadm'
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:48.917+0000
> > > 7f2663400700 -1 mgr
> > > > > > > > > > > load Traceback
> > > > > > > > > > > > > > > > (most recent call last):
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > "/usr/share/ceph/mgr/cephadm/module.py", line 470,
> > > > > > > > > > > in __init__
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > self.upgrade = CephadmUpgrade(self)
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > "/usr/share/ceph/mgr/cephadm/upgrade.py", line 112,
> > > > > > > > > > > in __init__
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > self.upgrade_state: Optional[UpgradeState] =
> > > > > > > > > > > > > > > > UpgradeState.from_json(json.loads(t))
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > "/usr/share/ceph/mgr/cephadm/upgrade.py", line 93,
> > > > > > > > > > > in from_json
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > return cls(**c)
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > TypeError: __init__() got an unexpected
> > > keyword argument
> > > > > > > > > > > > > 'daemon_types'
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > Apr 05 20:33:48 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:48.918+0000 7f2663400700
> > > > > > > > > > > > > > > > -1
> > > > > > > > > mgr operator()
> > > > > > > > > > > > > > > > Failed to run module in active mode ('cephadm')
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:49.273+0000
> > > 7f2663400700 -1 mgr
> > > > > > > > > > > load Failed to
> > > > > > > > > > > > > > > > construct class in 'snap_schedule'
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:49.273+0000
> > > 7f2663400700 -1 mgr
> > > > > > > > > > > load Traceback
> > > > > > > > > > > > > > > > (most recent call last):
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > > > > > "/usr/share/ceph/mgr/snap_schedule/module.py", line 38,
> > > > > > > > > > > > > in __init__
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > self.client = SnapSchedClient(self)
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > > > > > > > > >
> > > "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line
> > > > > > > > > > > > > > > > 158, in __init__
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > with self.get_schedule_db(fs_name) as conn_mgr:
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > File
> > > > > > > > > > >
> > > "/usr/share/ceph/mgr/snap_schedule/fs/schedule_client.py", line
> > > > > > > > > > > > > > > > 192, in get_schedule_db
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > db.executescript(dump)
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > sqlite3.OperationalError: table schedules
> > > already exists
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > Apr 05 20:33:49 cn03.ceph.xyz.corp
> > > > > > > > > > > > > > > >
> > > > > > > > >
> > > ceph-95f49c1c-b1e8-11ee-b5d0-0cc47a8f35c1-mgr-cn03-negzvb[307291]:
> > > > > > > > > > > > > > > > debug 2025-04-05T20:33:49.274+0000 7f2663400700
> > > > > > > > > > > > > > > > -1
> > > > > > > > > mgr operator()
> > > > > > > > > > > > > > > > Failed to run module in active mode
> > > ('snap_schedule')
> > > > > > > > > > > > > > > > ---snip---
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Your cluster seems to be in an error
> > > state (ceph -s)
> > > > > > > > > because of an
> > > > > > > > > > > > > > > > unknown PG. It's recommended to have a healthy
> > > > > > > cluster before
> > > > > > > > > > > > > > > > attemping an upgrade. It's possible that
> > > these errors
> > > > > > > > > > > come from the
> > > > > > > > > > > > > > > > not upgraded MGR, I'm not sure.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Since the upgrade was only successful for
> > > two MGRs, I
> > > > > > > > > am thinking
> > > > > > > > > > > > > > > > about downgrading both MGRs back to
> > > 16.2.15, then retry
> > > > > > > > > > > an upgrade to
> > > > > > > > > > > > > > > > a newer version, either 17.2.8 or 18.2.4.
> > > I haven't
> > > > > > > > > checked the
> > > > > > > > > > > > > > > > snap_schedule error yet, though. Maybe
> > > someone else knows
> > > > > > > > > > > > > that already.
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > > >
>
>
>
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-le...@ceph.io
