I think thinkgs are working as they were before again. Bob Proulx wrote: > > Is this related to migration to the new VM? > > Almost certainly it must be related. But when I test uploading to a > test project I do see the web site updated with the changes.
Maybe yes and maybe no. The stuck process timestamps from about the time of the move. But the problem was entirely on www.gnu.org and it is a problem that has happened before. So maybe no. > The way this works is a little bit of a "Rube Goldberg" device. CVS > updates trigger the loginfo hook processing which is simply a curl hit > of a URL on the web server machine. A python script on the web server > then goes and updates the web site copy of the project. Actually the python script only touches a file in a directory indicating that a project was updated. Then a cron script uses flock on /tmp/update-cvs.lock to semaphore running update-cvs.sh every two minutes to do the update. It's update-cvs.sh that was stuck. > Can you point me to a specific repository that is seeing this > behavior? With help from bandali on IRC I started looking at this page which was not updated. https://www.gnu.org/software/recent-releases-include.html Yet the repository clearly had an update. http://web.cvs.savannah.gnu.org/viewvc/www/www/software/recent-releases-include.html?r1=1.973&r2=1.974 So then I started poking around on www.gnu.org to figure out where things were stuck. I found this process tree. wwwcvs 29893 29889 0 Jun20 ? 00:00:00 /usr/bin/flock -w2 -x /tmp/update-cvs.lock /usr/local/bin/update-cvs.sh wwwcvs 29896 29893 0 Jun20 ? 00:00:03 /bin/bash /usr/local/bin/update-cvs.sh wwwcvs 26857 29896 0 Jun20 ? 00:00:00 /usr/bin/cvs -d :pserver:anon...@cvs.savannah.gnu.org:/webcvs/pdsplibrary checkout -P pdsplibrary The last log line from it: Thu Jun 20 16:15:09 EDT 2019 pdsplibrary: Beginning checkout (type non-gnu) cvs2 checkout: Updating pdsplibrary ... cvs2 checkout: Updating pdsplibrary/libpdssn/refman/html It is Fri, 21 Jun 2019 18:21:55 -0600 now. So that one checkout has been running for 24+ hours? That seems to have been the problem as to why web pages are not updating. And this problem has happened before. There is another cronjob that runs once a day and sends an email to sysadmin opening a ticket saying the following when this happens. I remember Lisa setting that up for me on a previous occasion when this had happened before. /usr/local/bin/update-cvs.sh is stuck on gnu.org This is all on www.gnu.org which was not part of the migration. I killed process 26857 above since that was stuck. That allowed the update-cvs.sh to move on. Exiting in this case. Then the */2 cron to run it started again. This run it walked through the queued work. It is still running. But I think it will be able to do what it does now. Looking at this page I see it is updated now. https://www.gnu.org/software/recent-releases-include.html I am going to keep an eye on it until it is through running. It appears to be making progress through the queued checkout work. I am surprised by number of directories with work queued. Literally thousands because it was still over a thousand and reducing before I thought to look to see how many. Bob
