I checked case (B) that I added. First I created a new setup as described (to make sure no old modifications influence the verification) That worked, I saw the wait line on boot: [*** ] A start job is running for Wait for… to be Configured (29s / no limit)
After boot I had the wait time in the failed service: $ systemctl status systemd-networkd-wait-online ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-04-16 13:52:04 UTC; 15s ago Installing the version in proposed ... $ sudo apt install systemd=237-3ubuntu10.21 Reading package lists... Done Building dependency tree Reading state information... Done The following package was automatically installed and is no longer required: grub-pc-bin Use 'sudo apt autoremove' to remove it. The following additional packages will be installed: libnss-systemd libpam-systemd libsystemd0 Suggested packages: systemd-container The following packages will be upgraded: libnss-systemd libpam-systemd libsystemd0 systemd 4 upgraded, 0 newly installed, 0 to remove and 42 not upgraded. Need to get 3318 kB of archives. After this operation, 11.3 kB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 libnss-systemd amd64 237-3ubuntu10.21 [105 kB] Get:2 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 libpam-systemd amd64 237-3ubuntu10.21 [108 kB] Get:3 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 systemd amd64 237-3ubuntu10.21 [2901 kB] Get:4 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 libsystemd0 amd64 237-3ubuntu10.21 [204 kB] Fetched 3318 kB in 1s (4658 kB/s) (Reading database ... 60080 files and directories currently installed.) Preparing to unpack .../libnss-systemd_237-3ubuntu10.21_amd64.deb ... Unpacking libnss-systemd:amd64 (237-3ubuntu10.21) over (237-3ubuntu10.19) ... Preparing to unpack .../libpam-systemd_237-3ubuntu10.21_amd64.deb ... Unpacking libpam-systemd:amd64 (237-3ubuntu10.21) over (237-3ubuntu10.19) ... Preparing to unpack .../systemd_237-3ubuntu10.21_amd64.deb ... Unpacking systemd (237-3ubuntu10.21) over (237-3ubuntu10.19) ... Preparing to unpack .../libsystemd0_237-3ubuntu10.21_amd64.deb ... Unpacking libsystemd0:amd64 (237-3ubuntu10.21) over (237-3ubuntu10.19) ... Setting up libsystemd0:amd64 (237-3ubuntu10.21) ... Processing triggers for ureadahead (0.100.0-20) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Setting up systemd (237-3ubuntu10.21) ... Processing triggers for man-db (2.8.3-2ubuntu0.1) ... Processing triggers for dbus (1.12.2-1ubuntu1) ... Setting up libnss-systemd:amd64 (237-3ubuntu10.21) ... Setting up libpam-systemd:amd64 (237-3ubuntu10.21) ... Processing triggers for libc-bin (2.27-3ubuntu1) ... Rebooting: - bootup was fast - service works fine $ systemctl status systemd-networkd-wait-online ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: active (exited) since Tue 2019-04-16 13:54:20 UTC; 10s ago Thereby good for case B, once A is confirmed as well please set it to verified. ** Description changed: [impact] systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases: a) if configured to ignore the dhcp server's route b) the server provides no route Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out). This delays boot as well as any restart to systemd-networkd. The fix is backporting upstream commit [1] [1]: https://github.com/systemd/systemd/commit/223932c7 [test case] There are two ways to test this. A) make the system ignore Routes (slightly less realistic but easier to test) configure an interface using systemd-networkd: $ cat /etc/systemd/network/20-ens7.network [Match] Name=ens7 [Network] DHCP=ipv4 [DHCP] UseRoutes=false then reboot, and check: $ systemctl status systemd-networkd-wait-online ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago [...] Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured... Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3 Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured. - B) Make a dhcp server to not provide rules Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool: - $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=disco - $ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily + $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic + $ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily Create a isolated network: $ cat > isolate.xml << EOF <network> - <name>isolated</name> - <!-- <bridge name='virbriso1' stp='on' delay='0'/> --> - <ip address='192.168.251.1' netmask='255.255.255.0'> - <dhcp> - <range start='192.168.251.2' end='192.168.251.254'/> - </dhcp> - </ip> + <name>isolated</name> + <!-- <bridge name='virbriso1' stp='on' delay='0'/> --> + <ip address='192.168.251.1' netmask='255.255.255.0'> + <dhcp> + <range start='192.168.251.2' end='192.168.251.254'/> + </dhcp> + </ip> </network> EOF $ virsh net-define isolate.xml - $ virsh net-start isolate.xml + $ virsh net-start isolate Edit the guest and add that network: $ virsh shutdown bionic $ virsh edit bionic $ virsh edit bionic #add this: - <interface type='network'> - <source network='isolated'/> - <mac address='52:54:00:c1:69:08'/> - <model type='virtio'/> - <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> - </interface> + <interface type='network'> + <source network='isolated'/> + <mac address='52:54:00:c1:69:08'/> + <model type='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> + </interface> In the guest make the device use DHCP: Add this to /etc/netplan/50-cloud-init.yaml - ens7: - dhcp4: true - match: - macaddress: 52:54:00:c1:69:08 - set-name: ens7 + ens7: + dhcp4: true + match: + macaddress: 52:54:00:c1:69:08 + set-name: ens7 $ sudo netplan apply When rebooting the guest again it will fail ● systemd-networkd-wait-online.service - Wait for Network to be Configured - Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) - Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago - Docs: man:systemd-networkd-wait-online.service(8) - Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) - Main PID: 563 (code=exited, status=1/FAILURE) + Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) + Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago + Docs: man:systemd-networkd-wait-online.service(8) + Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) + Main PID: 563 (code=exited, status=1/FAILURE) Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured... Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3 Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured. - When working cases A and B will both have the service starting fast and happy. $ sudo systemctl status systemd-networkd-wait-online - Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago + Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3 [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7 - [regression potential] this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting. The problem is that without a route there will be no netlink even t that can be tapped. But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero). Therefore the change seems rather small, reviewable and safe. [other info] original description: -- root cause in systemd bug fixed in Commit 223932c786ada7f758a7b7878a6ad2dae0d1e5fb: https://github.com/systemd/systemd/pull/8728 Environment: Ubuntu 18.04 LTS amd64 systemd package version 237-3ubuntu10.9 How to trigger: 1. add netplan interface with dhcpv4 client enabled: enp0s8: addresses: [] dhcp4: true 2. configure dhcp server to NOT give a default route, or any route for that matter 3. reboot 4. systemd-networkd-wait-online will block until dhcp renew is triggered root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 Event loop failed: Connection timed out root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8 managing: enp0s3 ignoring: lo ignoring: enp0s8 How to fix: Backport upstream changes from systemd (see related systemd ticket). ** Description changed: [impact] systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases: a) if configured to ignore the dhcp server's route b) the server provides no route Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out). This delays boot as well as any restart to systemd-networkd. The fix is backporting upstream commit [1] [1]: https://github.com/systemd/systemd/commit/223932c7 [test case] There are two ways to test this. A) make the system ignore Routes (slightly less realistic but easier to test) configure an interface using systemd-networkd: $ cat /etc/systemd/network/20-ens7.network [Match] Name=ens7 [Network] DHCP=ipv4 [DHCP] UseRoutes=false then reboot, and check: $ systemctl status systemd-networkd-wait-online ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago [...] Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured... Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3 Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured. B) Make a dhcp server to not provide rules Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool: $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic $ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily Create a isolated network: $ cat > isolate.xml << EOF <network> <name>isolated</name> <!-- <bridge name='virbriso1' stp='on' delay='0'/> --> <ip address='192.168.251.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.251.2' end='192.168.251.254'/> </dhcp> </ip> </network> EOF $ virsh net-define isolate.xml - $ virsh net-start isolate + $ virsh net-start isolated Edit the guest and add that network: $ virsh shutdown bionic - $ virsh edit bionic $ virsh edit bionic #add this: <interface type='network'> <source network='isolated'/> <mac address='52:54:00:c1:69:08'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> In the guest make the device use DHCP: Add this to /etc/netplan/50-cloud-init.yaml ens7: dhcp4: true match: macaddress: 52:54:00:c1:69:08 set-name: ens7 $ sudo netplan apply When rebooting the guest again it will fail ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago Docs: man:systemd-networkd-wait-online.service(8) Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) Main PID: 563 (code=exited, status=1/FAILURE) Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured... Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3 Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured. When working cases A and B will both have the service starting fast and happy. $ sudo systemctl status systemd-networkd-wait-online Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3 [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7 [regression potential] this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting. The problem is that without a route there will be no netlink even t that can be tapped. But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero). Therefore the change seems rather small, reviewable and safe. [other info] original description: -- root cause in systemd bug fixed in Commit 223932c786ada7f758a7b7878a6ad2dae0d1e5fb: https://github.com/systemd/systemd/pull/8728 Environment: Ubuntu 18.04 LTS amd64 systemd package version 237-3ubuntu10.9 How to trigger: 1. add netplan interface with dhcpv4 client enabled: enp0s8: addresses: [] dhcp4: true 2. configure dhcp server to NOT give a default route, or any route for that matter 3. reboot 4. systemd-networkd-wait-online will block until dhcp renew is triggered root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 Event loop failed: Connection timed out root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8 managing: enp0s3 ignoring: lo ignoring: enp0s8 How to fix: Backport upstream changes from systemd (see related systemd ticket). -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1804478 Title: netplan dhcp interface with no default route causes systemd-networkd- wait-online to hang Status in systemd: Fix Released Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Bionic: Fix Committed Bug description: [impact] systemd-networkd, when configured to use ipv4 dhcp for an interface can hang. This triggers in two known cases: a) if configured to ignore the dhcp server's route b) the server provides no route Then systemd-networkd will hang waiting for the interface's configuration to complete (until it times out). This delays boot as well as any restart to systemd-networkd. The fix is backporting upstream commit [1] [1]: https://github.com/systemd/systemd/commit/223932c7 [test case] There are two ways to test this. A) make the system ignore Routes (slightly less realistic but easier to test) configure an interface using systemd-networkd: $ cat /etc/systemd/network/20-ens7.network [Match] Name=ens7 [Network] DHCP=ipv4 [DHCP] UseRoutes=false then reboot, and check: $ systemctl status systemd-networkd-wait-online ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2019-04-08 23:59:26 UTC; 2min 59s ago [...] Apr 08 23:57:27 lp1804478 systemd[1]: Starting Wait for Network to be Configured... Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: managing: ens3 Apr 08 23:57:30 lp1804478 systemd-networkd-wait-online[593]: ignoring: lo Apr 08 23:59:26 lp1804478 systemd-networkd-wait-online[593]: Event loop failed: Connection timed out Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 08 23:59:26 lp1804478 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 08 23:59:26 lp1804478 systemd[1]: Failed to start Wait for Network to be Configured. B) Make a dhcp server to not provide rules Prepare a Ubuntu Bionic Guest under libvirt, e.g. using uvtool: $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=bionic $ uvt-kvm create --password ubuntu bionic arch=amd64 release=bionic label=daily Create a isolated network: $ cat > isolate.xml << EOF <network> <name>isolated</name> <!-- <bridge name='virbriso1' stp='on' delay='0'/> --> <ip address='192.168.251.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.251.2' end='192.168.251.254'/> </dhcp> </ip> </network> EOF $ virsh net-define isolate.xml $ virsh net-start isolated Edit the guest and add that network: $ virsh shutdown bionic $ virsh edit bionic #add this: <interface type='network'> <source network='isolated'/> <mac address='52:54:00:c1:69:08'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> In the guest make the device use DHCP: Add this to /etc/netplan/50-cloud-init.yaml ens7: dhcp4: true match: macaddress: 52:54:00:c1:69:08 set-name: ens7 $ sudo netplan apply When rebooting the guest again it will fail ● systemd-networkd-wait-online.service - Wait for Network to be Configured Loaded: loaded (/lib/systemd/system/systemd-networkd-wait-online.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-04-09 07:07:48 UTC; 1min 41s ago Docs: man:systemd-networkd-wait-online.service(8) Process: 563 ExecStart=/lib/systemd/systemd-networkd-wait-online (code=exited, status=1/FAILURE) Main PID: 563 (code=exited, status=1/FAILURE) Apr 09 07:05:48 bionic-dgx2 systemd[1]: Starting Wait for Network to be Configured... Apr 09 07:05:50 bionic-dgx2 systemd-networkd-wait-online[563]: managing: ens3 Apr 09 07:07:48 bionic-dgx2 systemd-networkd-wait-online[563]: Event loop failed: Connection timed out Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Main process exited, code=exited, status=1/FAILURE Apr 09 07:07:48 bionic-dgx2 systemd[1]: systemd-networkd-wait-online.service: Failed with result 'exit-code'. Apr 09 07:07:48 bionic-dgx2 systemd[1]: Failed to start Wait for Network to be Configured. When working cases A and B will both have the service starting fast and happy. $ sudo systemctl status systemd-networkd-wait-online Active: active (exited) since Tue 2019-04-09 07:17:16 UTC; 12s ago [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens3 [...] Apr 09 07:17:16 bionic-dgx2 systemd-networkd-wait-online[575]: managing: ens7 [regression potential] this alters how systemd-networkd works when it starts or restarts, specifically how it handles dhcp4; regressions would be around setting up interface(s) using dhcpv4, possibly failing to correctly configure dhcpv4 interface(s) or even failing internal assertions and exiting. The problem is that without a route there will be no netlink even t that can be tapped. But the change is only a 4 line change that does explicitly link->dhcp4_configured = true and calling link_check_ready() in dhcp4_address_handler() in case link_set_dhcp_routes() sent no netlink messages (dhcp4_messages is zero). Therefore the change seems rather small, reviewable and safe. [other info] original description: -- root cause in systemd bug fixed in Commit 223932c786ada7f758a7b7878a6ad2dae0d1e5fb: https://github.com/systemd/systemd/pull/8728 Environment: Ubuntu 18.04 LTS amd64 systemd package version 237-3ubuntu10.9 How to trigger: 1. add netplan interface with dhcpv4 client enabled: enp0s8: addresses: [] dhcp4: true 2. configure dhcp server to NOT give a default route, or any route for that matter 3. reboot 4. systemd-networkd-wait-online will block until dhcp renew is triggered root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 Event loop failed: Connection timed out root@sensor1:~# /lib/systemd/systemd-networkd-wait-online --timeout=3 --ignore=enp0s8 managing: enp0s3 ignoring: lo ignoring: enp0s8 How to fix: Backport upstream changes from systemd (see related systemd ticket). To manage notifications about this bug go to: https://bugs.launchpad.net/systemd/+bug/1804478/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
