Seems to be OK now. Had to stop and start the service to get it going again.
On Thu, 24 Sep 2020 at 17:16, sebb <seb...@gmail.com> wrote: > > I've just discovered that the service also falls over on Whimsy vm5. > > It appeared to be OK because it was not being used, but curl killed it > the same as on vm6. > > I have just run the following on vm5 > > gem [un]install eventmachine > > and curl no longer kills the process, so it's looking good > > I'll do the same on vm6 > > On Thu, 24 Sep 2020 at 16:53, sebb <seb...@gmail.com> wrote: > > > > On Thu, 24 Sep 2020 at 16:36, Sam Ruby <ru...@intertwingly.net> wrote: > > > > > > On Thu, Sep 24, 2020 at 11:12 AM sebb <seb...@gmail.com> wrote: > > > > > > > > On Thu, 24 Sep 2020 at 16:06, Sam Ruby <ru...@intertwingly.net> wrote: > > > > > > > > > > On Thu, Sep 24, 2020 at 10:58 AM sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > I tried running the following as root: > > > > > > > > > > > > /usr/local/bin/ruby /srv/whimsy/www/board/agenda/daemon/wss.rb -p > > > > > > 34243 > > > > > > > > > > > > This falls over when contacted by: > > > > > > > > > > > > curl localhost:34243 > > > > > > > > > > > > However, curiously it does not fall over when run as myself or > > > > > > www-data > > > > > > > > > > > > One obvious difference is that only root has access to > > > > > > /etc/letsencrypt/live/*/ > > > > > > which is used to set options.privkey and options.chain. > > > > > > > > > > Perhaps consistent with the theory that the gem is not compiled with > > > > > access to the encryption libraries. > > > > > > > > The letsencrypt certificates won't be available on initial Puppet load > > > > - maybe that affects the compilation options? > > > > > > > > Worth trying to reinstall now that the certs are up and running. > > > > > > While anything is possible, I doubt the existence of the certs matters > > > at install time. At install time, even if the certs were installed, > > > it wouldn't know which ones you were intending to use. The paths to > > > the certs are only provided at runtime. > > > > > > Again, my theory is that the puppet installation requests that both > > > libssl-dev and eventmachine need to be installed, but does not specify > > > in which order these packages need to be installed. And in this case, > > > the order is important. If eventmachine is installed first, it will > > > be installed without encryption support. If libssl-dev is installed > > > first, eventmachine will be installed with encryption support. > > > > Could be. > > > > libssl-dev is not mentioned in the whimsy5 puppet packages, but AFAICT > > it is installed. > > Maybe it comes pre-loaded in Ubuntu 18.04 which would explain how it > > worked there. > > > > I think we should try re-installing eventmachine. > > If that works, note the work-round until such time as the cause can be > > identified and fixed. > > That would require quite a lot of experimenting with Puppet and Docker... > > > > BTW, I'm not sure what you mean by the agenda console. > > > > > - Sam Ruby > > > > > > > > If you run without enabling options.privkey / options.chain, things > > > > > will work. If you enable those options, things will fail. > > > > > > > > > > - Sam Ruby > > > > > > > > > > > On Thu, 24 Sep 2020 at 15:30, sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > The login message said to reboot, so I have done this. > > > > > > > > > > > > > > However it does not appear to have helped. > > > > > > > > > > > > > > On Thu, 24 Sep 2020 at 14:28, sebb <seb...@gmail.com> wrote: > > > > > > > > > > > > > > > > On Thu, 24 Sep 2020 at 14:14, Sam Ruby <ru...@intertwingly.net> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > If you go into the board agenda tool and open the console, > > > > > > > > > you will > > > > > > > > > see messages that it can't connect to the websocket. Things > > > > > > > > > appear > > > > > > > > > to mostly be set up correctly: > > > > > > > > > > > > > > > > > > $ grep websocket > > > > > > > > > /etc/apache2/sites-available/10-whimsy-vm-443.conf > > > > > > > > > ProxyPass "/board/agenda/websocket/" "wss://localhost:34234/" > > > > > > > > > > > > > > > > > > $ sudo lsof -i :34234 > > > > > > > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > > > > > > > > > ruby 3430562 root 16u IPv4 160482206 0t0 TCP > > > > > > > > > *:34234 (LISTEN) > > > > > > > > > > > > > > > > > > $ sudo systemctl status board-agenda-websocket.service > > > > > > > > > ● board-agenda-websocket.service - Whimsy Board Agenda > > > > > > > > > WebSocket service > > > > > > > > > Loaded: loaded > > > > > > > > > (/etc/systemd/system/board-agenda-websocket.service; static; > > > > > > > > > vendor > > > > > > > > > preset: enabled) > > > > > > > > > Active: active (running) since Thu 2020-09-24 12:56:18 > > > > > > > > > UTC; 9s ago > > > > > > > > > Main PID: 3430562 (ruby) > > > > > > > > > Tasks: 8 (limit: 9541) > > > > > > > > > Memory: 35.5M > > > > > > > > > CGroup: /system.slice/board-agenda-websocket.service > > > > > > > > > └─3430562 /usr/local/bin/ruby > > > > > > > > > /srv/whimsy/www/board/agenda/daemon/wss.rb > > > > > > > > > > > > > > > > > > Sep 24 12:56:18 whimsy-vm6 systemd[1]: Started Whimsy Board > > > > > > > > > Agenda > > > > > > > > > WebSocket service. > > > > > > > > > > > > > > > > > > Note, however, how long the websocket has been running (9s). > > > > > > > > > This > > > > > > > > > indicates that it is dying and was restarted. Looking at the > > > > > > > > > syslog > > > > > > > > > confirms this: > > > > > > > > > > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 ruby[3435205]: terminate called > > > > > > > > > after > > > > > > > > > throwing an instance of 'std::runtime_error' > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 ruby[3435205]: what(): > > > > > > > > > Encryption not > > > > > > > > > available on this event-machine > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > > board-agenda-websocket.service: > > > > > > > > > Main process exited, code=dumped, status=6/ABRT > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > > board-agenda-websocket.service: > > > > > > > > > Failed with result 'core-dump'. > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: > > > > > > > > > board-agenda-websocket.service: > > > > > > > > > Scheduled restart job, restart counter is at 17895. > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: Stopped Whimsy Board > > > > > > > > > Agenda > > > > > > > > > WebSocket service. > > > > > > > > > Sep 24 13:09:07 whimsy-vm6 systemd[1]: Started Whimsy Board > > > > > > > > > Agenda > > > > > > > > > WebSocket service. > > > > > > > > > > > > > > > > > > Random googling turned up: > > > > > > > > > > > > > > > > > > https://github.com/huginn/huginn/issues/386 > > > > > > > > > > > > > > > > > > Perhaps it is a puppet sequence error, and the eventmachine > > > > > > > > > gem needs > > > > > > > > > to be installed AFTER libssl-dev? > > > > > > > > > > > > > > > > Looks like the code is working on vm5, which has a very similar > > > > > > > > Puppet setup. > > > > > > > > The main differences are some software versions. > > > > > > > > > > > > > > > > > - Sam Ruby
