Hi Bob, everything seems to be working fine now!
Many many thanks for all your (Werner and all savannah hackers) efforts on put that on place again and thanks for such a detailed explanation. It was amazing. Thanks once more and good luck with the migration! Marco Benatto Red Hat Product Security On Fri, Dec 13, 2019 at 3:46 PM Bob Proulx <b...@proulx.com> wrote: > > I believe the attachments should all be available again. Very sorry > for the disruption. The attachment directory was recently moved onto > an nfs mounted volume and that volume was not mounted when you found > those attachments inaccessible. The volume is mounted again now and > therefore all of the attachment files should be accessible again. > > Long details follow, ignore or skim as desired: > > We are in the process of upgrading the web UI frontend system to the > next version of the OS. Doing this by cloning the system, then > upgrading the clone, then testing the newly upgraded clone off to the > side so that we don't thrash the production frontend too much while we > work the bugs out of the new system. > > The attachments previously were directly on the local system. As you > can imagine the clone would have a separate copy of all of the > attachments. Then whichever system is in use and in testing would > then have a "split brain" problem of new attachments might get added > to one or the other but not both. That does not work very well if we > switch in an upgraded frontend if it had a separate local directory > attachment storage area. > > In order to simplify the upgrade I moved the attachment area to an NFS > mounted location where it would be shared between both the old and new > frontend servers. This is similar to what is done on the other > systems. That allows multiple systems to share access to the same > storage area. It allows both the old and new web UI frontend system > to share access to the attachments directory. One shared brain for > all. No split brains. > > But the old OS apparently has a known buggy problem with NFS mounts > timing out while looking for a kerberus daemon. So we for a short > time are dealing with an old problem on the old OS that is going away. > For details on that I include at the bottom some references I found to > the problem. All related to running the rpc-gssd service, used for > kerberos authentication. We aren't running kerberus and are affected > by this old bug which has been fixed on the newer OS version. So this > is just a temporary problem on the old system while we set up the new > one and will disappear entirely when we switch on the new system. > > At the time I observed the long delay in the client mounting the nfs > volume from the server. But after the minute or so timeout the mount > proceeded okay and subsequent access was okay. I did not think this > would be a real problem. > > And then the old frontend happened to be rebooted for an independent > need. While doing something unrelated I embarrassingly overwrote the > authorized_key file I used to access the system. Sigh. I had to have > FSF sysadmin reboot the system, mount the disk, restore the file, and > boot it back up again. Ian rescued it. That was an unexpected > reboot. And the NFS mount did not mount at boot time due to the bug > in the old OS! > > That made all of the attachments appear to have disappeared because > the shared NFS directory as not mounted. I am sorry. I knew the > system had gotten rebooted but I was distracted due to having cut off > my own access to the system and it slipped my mind that the NFS mount > might fail when it booted. I expected it would mount after a longer > than normal boot delay but apparently it is too long and fails at boot > time to mount. As soon as we can switch to the new OS version this > entire issue of nfs mount timing out disappears. > > Due to an Octave report on this problem (gack, everyone was affected) > I mounted the directory manually yesterday. And therefore all of the > attachments should be showing up again. Hopefully we will not be > rebooting again before we switch over to the new OS version. > > Again sincere apologies for the disruption! :-( > > Bob > > References on the nfs problem: > > https://bbs.archlinux.org/viewtopic.php?id=168317 > https://bugzilla.redhat.com/show_bug.cgi?id=1001934 > https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/1270445 > > https://serverfault.com/questions/631970/since-upgrading-nfs-and-ubuntu-to-14-04-all-nfs-mounts-are-failing > > > Marco Benatto wrote: > > Hi Werner and Ineiev, > > > > thank you very much for your help on this! > > > > Much appreciated! > > > > Marco Benatto > > Red Hat Product Security > > > > On Fri, Dec 13, 2019 at 4:57 AM Ineiev <ine...@gnu.org> wrote: > > > > > > On Fri, Dec 13, 2019 at 06:59:12AM +0100, Werner LEMBERG wrote: > > > > > > > > > > Generally, they don't disappear unless users remove them. > > > > > > > > OK, thanks for confirmation. With 'removing' you mean pressing the > > > > trashbin button, right? > > > > > > Yes, exactly. > > > > > > > I'm quite sure that this didn't happen. > > > > > > Confirmed. > > > > > > > >> Savannah hackers, do you know what's going on? What has happened > > > > >> to the three files attached to the above bug report? > > > > > > > > > > All attachments moved to a different directory on the server in the > > > > > process of upgrading its operating system, I'm not aware of full > > > > > details. > > > > > > > > Maybe something happened during this upgrade? > > > > > > Yes; in fact, the upgrade has not finished yet. > > > > > > > Is it possible to > > > > restore the links to those files in the bug report? > > > > > > Sure, we'll restore access to all files ASAP. > > > >
