Public bug reported: Last night, unattended-upgrades upgraded the erlang packages on our RabbitMQ server from 1:18.3-dfsg-1ubuntu3 to 1:18.3-dfsg-1ubuntu3.1. erlang-base's prerm script successfully found and stopped rabbitmq- server, and the postinst script tried to start it again:
# From /var/log/apt/term.log, same in /var/log/unattended-upgrades/unattended-upgrades-dpkg.log #... Preparing to unpack .../erlang-base_1%3a18.3-dfsg-1ubuntu3.1_amd64.deb ... Searching for services which depend on erlang and should be stopped...found: rabbitmq-server. Stopping services which depend on erlang rabbitmq-server: stopping...done. Services stopped successfully. Killing epmd...it is not running. Unpacking erlang-base (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ... Preparing to unpack .../erlang-nox_1%3a18.3-dfsg-1ubuntu3.1_all.deb ... Unpacking erlang-nox (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ... Processing triggers for man-db (2.7.5-1) ... Setting up erlang-base (1:18.3-dfsg-1ubuntu3.1) ... Searching for services which depend on erlang and should be started...found: rabbitmq-server. Starting services which depend on erlang rabbitmq-server: starting...done. Services started successfully. #... I looked into erlang-base.prerm and erlang-base.postinst and ran the relevant snippets manually, and it looks like they would have used "invoke-rc.d": invoke-rc.d rabbitmq-server stop > /dev/null 2>&1 # from erlang-base.prerm invoke-rc.d rabbitmq-server start > /dev/null 2>&1 # from erlang-base.postinst But while the stop command succeeded, the start command did not, and there isn't any evidence in the logs that it was ever issued. This is the reason why I think this is a bug in the erlang package (or possibly invoke-rc.d/init-system-helpers?), and not in rabbitmq-server: # journalctl Feb 16 06:30:03 rabbit-b systemd[1]: Starting Daily apt upgrade and clean activities... Feb 16 06:30:48 rabbit-b systemd[1]: Reloading. Feb 16 06:30:49 rabbit-b systemd[1]: Started ACPI event daemon. Feb 16 06:30:49 rabbit-b systemd[1]: Stopping RabbitMQ Messaging Server... Feb 16 06:30:50 rabbit-b rabbitmq[6596]: Stopping and halting node 'rabbit@rabbit-b' ... Feb 16 06:30:51 rabbit-b systemd[1]: Stopped RabbitMQ Messaging Server. Feb 16 06:30:52 rabbit-b systemd[1]: Reloading. Feb 16 06:30:52 rabbit-b systemd[1]: Started ACPI event daemon. Feb 16 06:30:54 rabbit-b systemd[1]: Reloading. Feb 16 06:30:54 rabbit-b systemd[1]: Started ACPI event daemon. # ... and the next log entry is from after unattended-upgrades had ended at 06:30:56 # /var/log/rabbitmq/rab...@rabbit-b.log =INFO REPORT==== 16-Feb-2018::06:30:50 === Stopping RabbitMQ #... lots of AMQP connection closures =INFO REPORT==== 16-Feb-2018::06:30:51 === Halting Erlang VM # ... and the next log entry is from 2h later when we manually started RabbitMQ again This is Ubuntu 16.04.2 LTS with all security patches and the latest rabbitmq-server (3.5.7-1ubuntu0.16.04.2) PS: Where are the sources for erlang-base.prerm and erlang- base.postinst? https://packages.debian.org/source/stretch/erlang points to http://anonscm.debian.org/viewvc/pkg-erlang/erlang/trunk/, but that link is dead ** Affects: erlang (Ubuntu) Importance: Undecided Status: New ** Description changed: Last night, unattended-upgrades upgraded the erlang packages on our RabbitMQ server from 1:18.3-dfsg-1ubuntu3 to 1:18.3-dfsg-1ubuntu3.1. erlang-base's prerm script successfully found and stopped rabbitmq- server, and the postinst script tried to start it again: # From /var/log/apt/term.log, same in /var/log/unattended-upgrades/unattended-upgrades-dpkg.log #... Preparing to unpack .../erlang-base_1%3a18.3-dfsg-1ubuntu3.1_amd64.deb ... Searching for services which depend on erlang and should be stopped...found: rabbitmq-server. Stopping services which depend on erlang - rabbitmq-server: stopping...done. + rabbitmq-server: stopping...done. Services stopped successfully. Killing epmd...it is not running. Unpacking erlang-base (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ... Preparing to unpack .../erlang-nox_1%3a18.3-dfsg-1ubuntu3.1_all.deb ... Unpacking erlang-nox (1:18.3-dfsg-1ubuntu3.1) over (1:18.3-dfsg-1ubuntu3) ... Processing triggers for man-db (2.7.5-1) ... Setting up erlang-base (1:18.3-dfsg-1ubuntu3.1) ... Searching for services which depend on erlang and should be started...found: rabbitmq-server. Starting services which depend on erlang - rabbitmq-server: starting...done. + rabbitmq-server: starting...done. Services started successfully. #... + I looked into erlang-base.prerm and erlang-base.postinst and ran the + relevant snippets manually, and it looks like they would have used + "invoke-rc.d": - I looked into erlang-base.prerm and erlang-base.postinst and ran the relevant snippets manually, and it looks like they would have used "invoke-rc.d": - - invoke-rc.d rabbitmq-server stop > /dev/null 2>&1 # from erlang-base.prerm - invoke-rc.d rabbitmq-server start > /dev/null 2>&1 # from erlang-base.postinst - + invoke-rc.d rabbitmq-server stop > /dev/null 2>&1 # from erlang-base.prerm + invoke-rc.d rabbitmq-server start > /dev/null 2>&1 # from erlang-base.postinst But while the stop command succeeded, the start command did not, and there isn't any evidence in the logs that it was ever issued. This is - the reason why I think this is a but in erlang (or possibly invoke-rc.d - /init-system-helpers?), and not in rabbitmq-server: + the reason why I think this is a bug in the erlang package (or possibly + invoke-rc.d/init-system-helpers?), and not in rabbitmq-server: # journalctl Feb 16 06:30:03 rabbit-b systemd[1]: Starting Daily apt upgrade and clean activities... Feb 16 06:30:48 rabbit-b systemd[1]: Reloading. Feb 16 06:30:49 rabbit-b systemd[1]: Started ACPI event daemon. Feb 16 06:30:49 rabbit-b systemd[1]: Stopping RabbitMQ Messaging Server... Feb 16 06:30:50 rabbit-b rabbitmq[6596]: Stopping and halting node 'rabbit@rabbit-b' ... Feb 16 06:30:51 rabbit-b systemd[1]: Stopped RabbitMQ Messaging Server. Feb 16 06:30:52 rabbit-b systemd[1]: Reloading. Feb 16 06:30:52 rabbit-b systemd[1]: Started ACPI event daemon. Feb 16 06:30:54 rabbit-b systemd[1]: Reloading. Feb 16 06:30:54 rabbit-b systemd[1]: Started ACPI event daemon. # ... and the next log entry is from after unattended-upgrades had ended at 06:30:56 # /var/log/rabbitmq/rab...@rabbit-b.log - =INFO REPORT==== 16-Feb-2018::06:30:50 === + =INFO REPORT==== 16-Feb-2018::06:30:50 === Stopping RabbitMQ #... lots of AMQP connection closures =INFO REPORT==== 16-Feb-2018::06:30:51 === Halting Erlang VM # ... and the next log entry is from 2h later when we manually started RabbitMQ again + This is Ubuntu 16.04.2 LTS with all security patches and the latest + rabbitmq-server (3.5.7-1ubuntu0.16.04.2) - This is Ubuntu 16.04.2 LTS with all security patches and the latest rabbitmq-server (3.5.7-1ubuntu0.16.04.2) - - - PS: Where are the sources for erlang-base.prerm and erlang-base.postinst? https://packages.debian.org/source/stretch/erlang points to http://anonscm.debian.org/viewvc/pkg-erlang/erlang/trunk/, but that link is dead + PS: Where are the sources for erlang-base.prerm and erlang- + base.postinst? https://packages.debian.org/source/stretch/erlang points + to http://anonscm.debian.org/viewvc/pkg-erlang/erlang/trunk/, but that + link is dead -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1749959 Title: Upgrade of erlang-base left rabbitmq-server dead To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1749959/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
