On Mon, Oct 30, 2023 at 03:11:32PM +0100, Ole Holm Nielsen wrote: Hi Max & freinds, ... > Thanks so much for your fast response with a solution! I didn't know that > NetworkManager (falsely) claims that the network is online as soon as the > first interface comes up :-(
IIRC it is documented in the man page. > Your solution of a wait-for-interfaces Systemd service makes a lot of sense, > and I'm going to try it out. Actually there is no need for such a script since /lib/systemd/systemd-networkd-wait-online should be able to handle it. I.e. 'Exec=/lib/systemd/systemd-networkd-wait-online -i ib0:routable' or something like that should handle it. E.g. on my laptop the complete /etc/systemd/system/systemd-networkd-wait-online.service looks like this: ---schnipp--- [Unit] Description=Wait for Network to be Configured Documentation=man:systemd-networkd-wait-online.service(8) DefaultDependencies=no Conflicts=shutdown.target Requires=systemd-networkd.service After=systemd-networkd.service Before=network-online.target shutdown.target [Service] Type=oneshot ExecStart=/lib/systemd/systemd-networkd-wait-online -i eth0:routable -i wlan0:routable --any RemainAfterExit=yes [Install] WantedBy=network-online.target ---schnapp--- Have fun, jel. > Best regards, > Ole > > On 10/30/23 14:30, Max Rutkowski wrote: > > Hi, > > > > we're not using Omni-Path but also had issues with Infiniband taking too > > long and slurmd failing to start due to that. > > > > Our solution was to implement a little wait-for-interface systemd > > service which delays the network.target until the ib interface has come > > up. > > > > Our discovery was that the network-online.target is triggered by the > > NetworkManager as soon as the first interface is connected. > > > > I've put the solution we use on my GitHub: > > https://github.com/maxlxl/network.target_wait-for-interfaces > > > > You may need to do small adjustments, but it's pretty straight forward > -- > Ole Holm Nielsen > PhD, Senior HPC Officer > Department of Physics, Technical University of Denmark, > Fysikvej Building 309, DK-2800 Kongens Lyngby, Denmark > E-mail: ole.h.niel...@fysik.dtu.dk > Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/ > Mobile: (+45) 5180 1620 in > > general. > > > > > > Kind regards > > Max > > > > On 30.10.23 13:50, Ole Holm Nielsen wrote: > > > I'm fighting this strange scenario where slurmd is started before > > > the Infiniband/OPA network is fully up. The Node Health Check (NHC) > > > executed by slurmd then fails the node (as it should). This happens > > > only on EL8 Linux (AlmaLinux 8.8) nodes, whereas our CentOS 7.9 > > > nodes with Infiniband/OPA network work without problems. > > > > > > Question: Does anyone know how to reliably delay the start of the > > > slurmd Systemd service until the Infiniband/OPA network is fully up? > > > > > > Note: Our Infiniband/OPA network fabric is Omni-Path 100 Gbit/s, not > > > Mellanox IB. On AlmaLinux 8.8 we use the in-distro OPA drivers > > > since the CornelisNetworks drivers are not available for RHEL 8.8. > -- > Ole Holm Nielsen > PhD, Senior HPC Officer > Department of Physics, Technical University of Denmark, > Fysikvej Building 309, DK-2800 Kongens Lyngby, Denmark > E-mail: ole.h.niel...@fysik.dtu.dk > Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/ > Mobile: (+45) 5180 1620 > > > > > > The details: > > > > > > The slurmd service is started by the service file > > > /usr/lib/systemd/system/slurmd.service after the > > > "network-online.target" has been reached. > > > > > > It seems that NetworkManager reports "network-online.target" BEFORE > > > the Infiniband/OPA device ib0 is actually up, and this seems to be > > > the cause of our problems! > > > > > > Here are some important sequences of events from the syslog showing > > > that the network goes online before the Infiniband/OPA network > > > (hfi1_0 adapter) is up: > > > > > > Oct 30 13:01:40 d064 systemd[1]: Reached target Network is Online. > > > (lines deleted) > > > Oct 30 13:01:41 d064 slurmd[2333]: slurmd: error: health_check > > > failed: rc:1 output:ERROR: nhc: Health check failed: check_hw_ib: > > > No IB port is ACTIVE (LinkUp 100 Gb/sec). > > > (lines deleted) > > > Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0: 8051: Link up > > > Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0: > > > set_link_state: current GOING_UP, new INIT (LINKUP) > > > Oct 30 13:01:41 d064 kernel: hfi1 0000:4b:00.0: hfi1_0: physical > > > state changed to PHYS_LINKUP (0x5), phy 0x50 > > > > > > I tried to delay the NetworkManager "network-online.target" by > > > setting a wait on the ib0 device and reboot, but that seems to be > > > ignored: > > > > > > $ nmcli -p connection modify "System ib0" > > > connection.connection.wait-device-timeout 20 > > > > > > I'm hoping that other sites using Omni-Path have seen this and maybe > > > can share a fix or workaround? > > > > > > Of course we could remove the Infiniband check in Node Health Check > > > (NHC), but that would not really be acceptable during operations. > > > > > > Thanks for sharing any insights, > > > Ole > > > > > -- > > Max Rutkowski > > IT-Services und IT-Betrieb > > Tel.: +49 (0)331/6264-2341 > > E-Mail: max.rutkow...@gfz-potsdam.de > > ___________________________________ > > > > Helmholtz-Zentrum Potsdam > > *Deutsches GeoForschungsZentrum GFZ* > > Stiftung des öff. Rechts Land Brandenburg > > Telegrafenberg, 14473 Potsdam -- Otto-von-Guericke University http://www.cs.uni-magdeburg.de/ Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2 39106 Magdeburg, Germany Tel: +49 391 67 52768
