Public bug reported: Hello - we are seeing an issue on multiple Azure hosts where there is a long delay during bootup. This appears to be related to an apparmor issue with dhclient executed via cloud-init when /var is mounted noexec. Because /var is noexec, the original dhclient is executed rather than the copy in /var/tmp/cloud-init, which causes the AppArmor profile to be applied.
This prevents the instance from being able to record the DHCP lease information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents the instance from being able to obtain goalstate information, and with cloud-init 21.2-3 or later, results in an extended delay during boot due to a recent change in azure.py (https://github.com/canonical/cloud- init/pull/842). This issue does not occur in default Ubuntu installations (including the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud-init behaves differently, copying /usr/sbin/dhclient to /var/tmp/cloud-init /cloud-init-dhcp-xxxxx/dhclient when /var allows executables, and the apparmor profiles then do not apply to the copied executable. The syslog will show the following entry when the instance boots up: cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected: azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable')) With the timeouts in azure.py described above, the instance will not boot for around 20 minutes until all 240 connection attempts are completed. This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process: type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0 Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient /var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw, /var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw, This allows dhclient executed via cloud-init to write the dhclient.pid and dhcp.leases files to /var/tmp/cloud-init and the instance to boot normally. ** Affects: isc-dhcp (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to isc-dhcp in Ubuntu. https://bugs.launchpad.net/bugs/1934933 Title: cloud-init dhclient apparmor denied with noexec on /var/tmp Status in isc-dhcp package in Ubuntu: New Bug description: Hello - we are seeing an issue on multiple Azure hosts where there is a long delay during bootup. This appears to be related to an apparmor issue with dhclient executed via cloud-init when /var is mounted noexec. Because /var is noexec, the original dhclient is executed rather than the copy in /var/tmp/cloud-init, which causes the AppArmor profile to be applied. This prevents the instance from being able to record the DHCP lease information to /var/tmp/cloud-init/cloud-init-dhcp-*, which prevents the instance from being able to obtain goalstate information, and with cloud-init 21.2-3 or later, results in an extended delay during boot due to a recent change in azure.py (https://github.com/canonical /cloud-init/pull/842). This issue does not occur in default Ubuntu installations (including the Ubuntu 20.04 default Azure image), as the dhcp.py script in cloud- init behaves differently, copying /usr/sbin/dhclient to /var/tmp /cloud-init/cloud-init-dhcp-xxxxx/dhclient when /var allows executables, and the apparmor profiles then do not apply to the copied executable. The syslog will show the following entry when the instance boots up: cloud-init[820]: 2021-07-07 14:50:40,661 - dhcp.py[WARNING]: dhclient did not produce expected files: dhcp.leases, dhclient.pid The cloud-init.log file will show this entry when this issue is occurring. Since the instance has no IP address at this stage of the boot process, an unreachable network is to be expected: azure.py[DEBUG]: Failed HTTP request with Azure endpoint http://168.63.129.16/machine/?comp=goalstate during attempt 240 with exception: HTTPConnectionPool(host='168.63.129.16', port=80): Max retries exceeded with url: /machine/?comp=goalstate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd9e89ee190>: Failed to establish a new connection: [Errno 101] Network is unreachable')) With the timeouts in azure.py described above, the instance will not boot for around 20 minutes until all 240 connection attempts are completed. This is logged in /var/log/audit/audit.log, showing that the dhclient process executed from cloud-init is unable to write the dhclient.pid and dhcp.leases files that are needed to continue the datasource retrieval process: type=AVC msg=audit(1625678140.496:1898): apparmor="DENIED" operation="open" profile="/{,usr/}sbin/dhclient" name="/proc/8537/task/8540/comm" pid=8537 comm="dhclient" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0 Adding the following file resolves the issue: /etc/apparmor.d/local/sbin.dhclient /var/tmp/cloud-init/cloud-init-dhcp-*/dhclient.pid lrw, /var/tmp/cloud-init/cloud-init-dhcp-*/dhcp.leases lrw, This allows dhclient executed via cloud-init to write the dhclient.pid and dhcp.leases files to /var/tmp/cloud-init and the instance to boot normally. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/isc-dhcp/+bug/1934933/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
