I reviewed the upgrade suites. There were some infrastructure problems that affected a significant number of the results, which I think we should get fixed before we approve the upgrades.
@David Galloway <[email protected]>, the main issue is that we are missing packages/containers for Quincy and Pacific. See details in https://tracker.ceph.com/issues/75034. The reef point-to-point tests did not have any infrastructure failures, but there were two new bugs that I raised for RBD and the Dashboard. @Ilya Dryomov <[email protected]> and @[email protected] <[email protected]>, can you take a look? *upgrade:quincy-x* 1. https://tracker.ceph.com/issues/75034 - Missing packages and container images in upgrade tests - (Infrastructure) *upgrade:pacific-x* 1. https://tracker.ceph.com/issues/75034 - Missing packages and container images in upgrade tests - (Infrastructure) *upgrade:reef-p2p* 1. https://tracker.ceph.com/issues/75037 - src/test/pybind/test_rbd.py::test_remove_canceled fails from coredump - (rbd) 2. https://tracker.ceph.com/issues/75038 - Dashboard module and Prometheus module fail to load during upgrade test - (Dashboard) On Mon, Feb 16, 2026 at 4:40 AM Venky Shankar via ceph-users < [email protected]> wrote: > Hi Yuri, > > On Thu, Feb 12, 2026 at 9:30 PM Yuri Weinstein via ceph-users > <[email protected]> wrote: > > > > Still seeking approvals for: > > > > (smoke - FAILED, was reviewed and approved by Laura and Patrick, see > Slack > > thread for details) > > rados - FAILED, Afreen, Travis? ( Laura approved for core) > > > > rgw - FAILED, Adam E approved? > > > > (rbd, krbd - FAILED, Ilya approved) > > fs - FAILED, Venky approved? > > I wasn't able to finish reviewing the run and teuthology results are > unavailable right now due to the lab outage. > > > orch - FAILED, Adam K approved? > > > > upgrade/pacific-x (reef) - FAILED Laura, approved? > > upgrade/quincy-x (reef) - FAILED Laura, approved? > > upgrade/reef-p2p - FAILED Laura, approved? > > > > (powercycle - FAILED Brad, approved) > > ceph-volume - Approved Guillaume? > > > > On Wed, Feb 11, 2026 at 12:10 PM Laura Flores via Dev <[email protected]> > wrote: > > > > > The rados suite is approved: > > > > https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackercephcomissues73906note-1 > > > > > > On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <[email protected]> > wrote: > > > > > >> Powercycle approved. > > >> > > >> On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <[email protected]> > > >> wrote: > > >> > > >>> > > >>> > > >>> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <[email protected]> > > >>> wrote: > > >>> > > >>>> > > >>>> > > >>>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < > > >>>>> [email protected]> wrote: > > >>>>> > > >>>>> > We are continuing on this release. > > >>>>> > > > >>>>> > > > >>>>> > powercycle - FAILED Brad, approved? > > >>>>> > > > >>>>> > > >>>>> Unfortunately, I don't see any logs anywhere where I have access to > > >>>>> them so > > >>>>> I have no way of checking this? > > >>>>> > > >>>> > > >>>> OK, I have the latest logs. The three failing jobs appear to have > all > > >>>> timed out after running for about six hours. At this time I don't > know why > > >>>> but I continue to investigate. > > >>>> > > >>> > > >>> One point of commonality of the three tests that failed is that they > are > > >>> all running tests using ceph fuse. I'll reach out to the FS guys to > see if > > >>> they can help me get to the bottom of these failures. > > >>> > > >>> > > >>>> > > >>>> > > >>>>> > > >>>>> > > >>>>> > ceph-volume - Approved Guillaume? > > >>>>> > > > >>>>> > Patrick was reviewing some tests, please see his comments in the > same > > >>>>> > tracker. > > >>>>> > TIA > > >>>>> > > > >>>>> > > > >>>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < > > >>>>> > [email protected]> wrote: > > >>>>> > > > >>>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev < > > >>>>> [email protected]> > > >>>>> > > wrote: > > >>>>> > > > > > >>>>> > > > This test branch was built loner then 2.5 weeks ago > > >>>>> > > > I have to rebuild it first and then will schedule krbd > > >>>>> > > > > >>>>> > > krbd approved based on > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testing-default-smithi/ > > >>>>> > > > > >>>>> > > > > >>>>> > > > >>>>> > https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testing-default-smithi/ > > >>>>> > > > > >>>>> > > Thanks, > > >>>>> > > > > >>>>> > > Ilya > > >>>>> > > > > >>>>> > > > > > >>>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < > > >>>>> > > [email protected]> wrote: > > >>>>> > > >> > > >>>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov < > [email protected] > > >>>>> > > > >>>>> > wrote: > > >>>>> > > >> > > > >>>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via > ceph-users > > >>>>> > > >> > <[email protected]> wrote: > > >>>>> > > >> > > > > >>>>> > > >> > > Hi Yuri, > > >>>>> > > >> > > > > >>>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < > > >>>>> > [email protected]> > > >>>>> > > wrote: > > >>>>> > > >> > > > > > >>>>> > > >> > > > Note: this is an attempt to test and publish post EOL > > >>>>> reef > > >>>>> > point > > >>>>> > > release > > >>>>> > > >> > > > > > >>>>> > > >> > > > Details of this release are summarized here: > > >>>>> > > >> > > > > > >>>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 > > >>>>> > > >> > > > Release Notes - TBD > > >>>>> > > >> > > > LRC upgrade - N/A > > >>>>> > > >> > > > > > >>>>> > > >> > > > distro => jammy focal centos9 windows bookworm > > >>>>> > > >> > > > > > >>>>> > > >> > > > Seeking approvals/reviews for: > > >>>>> > > >> > > > > > >>>>> > > >> > > > smoke - Laura approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? > Adam > > >>>>> King > > >>>>> > > approved? > > >>>>> > > >> > > > (rados is also running on build 2 == build 1 + > > >>>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's > > >>>>> request) > > >>>>> > > >> > > > > > >>>>> > > >> > > > rgw - Adam E approved? > > >>>>> > > >> > > > > > >>>>> > > >> > > > fs - Venky approved? > > >>>>> > > >> > > > > >>>>> > > >> > > There are lot many failures in the fs suite resembling > > >>>>> > > >> > > > > >>>>> > > >> > > Error mounting /var/lib/ceph: mount: > /var/lib/ceph: > > >>>>> wrong > > >>>>> > fs > > >>>>> > > >> > > type, bad option ... > > >>>>> > > >> > > > > >>>>> > > >> > > which is an issue with the testing kernel. Adding Alex > who > > >>>>> was > > >>>>> > > looking > > >>>>> > > >> > > into this issue. > > >>>>> > > >> > > > > >>>>> > > >> > > As far as the run validation is concerned, it's risky to > > >>>>> ignore > > >>>>> > the > > >>>>> > > >> > > large number of failures. We have two options: > > >>>>> > > >> > > > > >>>>> > > >> > > - fix the testing kernel issue and rerun > > >>>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in > the > > >>>>> > > >> > > teuthology-suite schedule command) > > >>>>> > > >> > > > > >>>>> > > >> > > The latter would be using an older kernel branch, but I > > >>>>> think it > > >>>>> > > >> > > should suffice for this case. > > >>>>> > > >> > > > >>>>> > > >> > Hi Venky, > > >>>>> > > >> > > > >>>>> > > >> > With your report of the fs suite being affected as well, I > > >>>>> carved > > >>>>> > out > > >>>>> > > >> > some time to take a look. > > >>>>> > > >> > > > >>>>> > > >> > I think it's happening because XFS maintainers disabled > > >>>>> support for > > >>>>> > V4 > > >>>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 > > >>>>> kernel > > >>>>> > it's > > >>>>> > > >> > no longer compiled in by default. The proper fix would > be to > > >>>>> create > > >>>>> > > an > > >>>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to > > >>>>> try to > > >>>>> > > >> > achieve that. We can also re-enable V4 on-disk format in > our > > >>>>> kernel > > >>>>> > > >> > builds as a workaround. > > >>>>> > > >> > > > >>>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > >>>>> > > >> > > >>>>> > > >> This should be fixed now. Yuri, please schedule a rerun for > > >>>>> krbd. > > >>>>> > > >> > > >>>>> > > >> Thanks, > > >>>>> > > >> > > >>>>> > > >> Ilya > > >>>>> > > >> _______________________________________________ > > >>>>> > > >> ceph-users mailing list -- [email protected] > > >>>>> > > >> To unsubscribe send an email to [email protected] > > >>>>> > > > > > >>>>> > > > _______________________________________________ > > >>>>> > > > Dev mailing list -- [email protected] > > >>>>> > > > To unsubscribe send an email to [email protected] > > >>>>> > > _______________________________________________ > > >>>>> > > ceph-users mailing list -- [email protected] > > >>>>> > > To unsubscribe send an email to [email protected] > > >>>>> > > > > >>>>> > _______________________________________________ > > >>>>> > ceph-users mailing list -- [email protected] > > >>>>> > To unsubscribe send an email to [email protected] > > >>>>> > > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Cheers, > > >>>>> Brad > > >>>>> _______________________________________________ > > >>>>> ceph-users mailing list -- [email protected] > > >>>>> To unsubscribe send an email to [email protected] > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Cheers, > > >>>> Brad > > >>>> > > >>> > > >>> > > >>> -- > > >>> Cheers, > > >>> Brad > > >>> _______________________________________________ > > >>> Dev mailing list -- [email protected] > > >>> To unsubscribe send an email to [email protected] > > >>> > > >> > > >> > > >> -- > > >> Cheers, > > >> Brad > > >> _______________________________________________ > > >> Dev mailing list -- [email protected] > > >> To unsubscribe send an email to [email protected] > > >> > > > > > > > > > -- > > > > > > Laura Flores > > > > > > She/Her/Hers > > > > > > Software Engineer, Ceph Storage <https://ceph.io> > > > > > > Chicago, IL > > > > > > [email protected] | [email protected] <[email protected]> > > > M: +17087388804 > > > > > > > > > _______________________________________________ > > > Dev mailing list -- [email protected] > > > To unsubscribe send an email to [email protected] > > > > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > > -- > Cheers, > Venky > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL [email protected] | [email protected] <[email protected]> M: +17087388804 _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
