I'm announcing an initial version 0.1 of an Infiniband topology tool
"slurmibtopology.sh" for Slurm.
I had to create a Slurm topology.conf file and needed an automated way
to get the correct node and switch Infiniband connectivity. The manual
page https://slurm.schedmd.com/topology.conf.html refers to an outdated
tool ib2slurm.
Inspired by the script in
https://unix.stackexchange.com/questions/255472/text-processing-building-a-slurm-topology-conf-file-from-ibnetdiscover-output
I decided to write a simpler and more understandable tool. It parses
the output of the OFED command "ibnetdiscover" and generates an initial
topology.conf file (which you may want to edit for readability).
Download the script https://ftp.fysik.dtu.dk/Slurm/slurmibtopology.sh
Example: Running this script on our Infiniband network:
# slurmibtopology.sh
Verify the Infiniband interface:
mlx4_0
Infiniband interface OK
Generate the Slurm topology.conf file for Infiniband switches:
# IB switch no. 1: MF0;mell02:IS5030/U1
SwitchName=ibsw1 Nodes=i[001-028]
# IB switch no. 2: MF0;mell03:IS5030/U1
SwitchName=ibsw2 Nodes=finbul,i[029-051],niflfs[1-2],niflopt1
# IB switch no. 3: MF0;mell01:SX6036/U1
SwitchName=ibsw3 Nodes=g[081-100,102-112],h[001-002]
# IB switch no. 4: MF0;mell04:SX6036/U1
# NOTICE: This switch has no attached nodes (empty hostlist)
SwitchName=ibsw4 Nodes=""
# IB switch no. 5: Mellanox 4036 # volt01
SwitchName=ibsw5 Nodes=g[005-008,013-014,016,021-024,029-032,037-040]
# IB switch no. 6: Mellanox 4036 # volt03
SwitchName=ibsw6 Nodes=g[045-048,053-056,061-064,069-072,077-080]
# IB switch no. 7: Mellanox 4036 # volt04
SwitchName=ibsw7 Nodes=g[041-044,049-052,057-060,065-068,073-076]
# IB switch no. 8: Mellanox 4036 # volt02
SwitchName=ibsw8 Nodes=g[001-004,009-012,017-020,025-028,033-036]
# Merging all switches in a top-level spine switch
SwitchName=spineswitch Switches=ibsw[1-8]
It would be great if other sites could test this tool on their
Infiniband network and report bugs or suggest improvements.
--
Ole Holm Nielsen
Department of Physics, Technical University of Denmark
