This is what I discovered: This systemd issue is caused because there are iscsi connections still logged in (not because of multipath, bonding, vlans, etc). One way of reproducing this is by simply editing the open-iscsi.service unit and disabling the logout script. It will, for me in all times, cause the same hang as we are talking to.
inaddy@iscsihang:~$ sudo systemctl edit --full open-iscsi.service ... ExecStop=/lib/open-iscsi/umountiscsi.sh ExecStop=/bin/sync #ExecStop=/lib/open-iscsi/logout-all.sh But of course no one is doing this. So, reading logout-all you will discover that it logs out all paths but it has some caveats: ISCSI_ROOT_KEEP_ALL_SESSIONS_AT_SHUTDOWN (from /etc/default/open-iscsi) and /run/open-iscsi/shutdown-keep-sessions would cause those iscsi sessions not to be logged out. Still, I wasn't using neither to reproduce so I found out something else. Disabling both systemd services: inaddy@iscsihang:~$ systemctl list-unit-files --full | grep iscsi iscsi.service disabled (alias for iscsid) iscsid.service disabled open-iscsi.service disabled I was still able to reproduce the issue: http://pastebin.ubuntu.com/25388348/ By issuing a logout when the iscsid daemon is gone. It looks like iscsid is lost after it has died/been killed on the sessions that were previously established. If you try to login again, it will be totally lost when trying to logout, causing leftovers (opened iscsi sessions) causing systemd to hang (even not having daemons running, in this case). I'm investigating: - Should systemd hang on those leftovers ? Since there are caveats that explicitly allow you to leave opened sessions, for / for example, I doubt it. - Does iscsid daemon contain a bug that causes sessions to remain un- mapped and opened if daemon is restarted without a logout of previously opened sessions ? Now that I'm able to reproduce this at will, I'll be able to answer those questions soon. Best, Rafael PS: Matt, I asked partnership program managers to reach out to understand better your future needs. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1569925 Title: Shutdown hang on 16.04 with iscsi targets To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1569925/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
