This is how this can be debugged with lxd: Change the reproducer script
to not delete the "working" containers, but leave them running. Then
booting the failed x-013 (or whichever it is) will fail reliably. "lxc
stop -f x-013" it, then "lxc config edit x-013" and add the following
keys:
config:
environment.SYSTEMD_LOG_LEVEL: debug
devices:
kmsg:
major: "1"
minor: "11"
path: /dev/kmsg
type: unix-char
This will enable verbose logging and provide a /dev/kmsg in the
container so that the container can actually log stuff to dmesg (it
fails too early for the journal to start). Then "dmesg -w" provides a
nice log tail. There we see:
[12235.886344] systemd-tmpfiles[54]: [/usr/lib/tmpfiles.d/var.conf:14]
Duplicate line for path "/var/log", ignoring.
[12235.891623] systemd-remount-fs[55]: /bin/mount for / exited with exit status
1.
[12235.909095] systemd-udevd[51]: inotify_init failed: Too many open files
[12235.909101] systemd-udevd[51]: error initializing inotify
[12235.909151] systemd-udevd[51]: failed to allocate manager object: Cannot
allocate memory
(repeats a few times). stracing lxd children also confirms that there
are a lot of EMFILE errors.
However, I stopped x-013 and rebooted x-012, which works again. The
EMFILE *also* happens there, so this is unrelated and just a red
herring. The important difference is that udev works there, so systemd
sees that the root device is already mounted/does not exist. Without
udev it just tries to mount it as it has no further information about
it.
As a workaround, remove or comment out the root fs line in /etc/fstab --
it's not needed at all in a container. Then the container boots, which
makes it more convenient to debug udev in it.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1602192
Title:
deploy 30 nodes on lxd, machines never leave pending
To manage notifications about this bug go to:
https://bugs.launchpad.net/juju-core/+bug/1602192/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
