Hi Damien,
Thanks for your positive feedback! I'll be posting an updated script
soon which works better for multilevel IB networks.
I would however warn anyone against using the output of
slurmibtopology.sh directly in an automated procedure. I recommend
strongly a manual step as well in a procedure like:
1. Use slurmibtopology.sh to generate topology.conf. It should show
correctly the leaf switches and their links to compute nodes.
2. Review your 2nd and 3rd level network topology as discussed in
https://slurm.schedmd.com/topology.html. Heed in particular this statement:
As a practical matter, listing every switch connection definitely results in a
slower scheduling algorithm for Slurm to optimize job placement. The
application performance may achieve little benefit from such optimization.
Listing the leaf switches with their nodes plus one top level switch should
result in good performance for both applications and Slurm.
3. In the generated topology.conf you should select only 1 top-level
switch and delete the others.
4. Copy the edited topology.conf to your cluster.
/Ole
On 05/08/2017 04:56 PM, Damien François wrote:
Hi
many thanks for the tools, it works flawlessly here.
I just patched to send the output that does not belong to the topology.conf to
stderr so I could simply redirect to topology.conf in an automated Slurm
install procedure
16c16
< echo Verify the Infiniband interface: >&2
---
echo Verify the Infiniband interface:
18c18
< if $IBSTAT -l >&2
---
if $IBSTAT -l
20c20
< echo Infiniband interface OK >&2
---
echo Infiniband interface OK
22c22
< echo Infiniband interface NOT OK >&2
---
echo Infiniband interface NOT OK
26c26
< cat <<EOF >&2
---
cat <<EOF
Sincerely
damien
On 07 May 2017, at 14:29, Ole Holm Nielsen <[email protected]> wrote:
I'm announcing an initial version 0.1 of an Infiniband topology tool
"slurmibtopology.sh" for Slurm.
I had to create a Slurm topology.conf file and needed an automated way to get
the correct node and switch Infiniband connectivity. The manual page
https://slurm.schedmd.com/topology.conf.html refers to an outdated tool
ib2slurm.
Inspired by the script in
https://unix.stackexchange.com/questions/255472/text-processing-building-a-slurm-topology-conf-file-from-ibnetdiscover-output
I decided to write a simpler and more understandable tool. It parses the output of the
OFED command "ibnetdiscover" and generates an initial topology.conf file (which
you may want to edit for readability).
Download the script https://ftp.fysik.dtu.dk/Slurm/slurmibtopology.sh
Example: Running this script on our Infiniband network:
# slurmibtopology.sh
Verify the Infiniband interface:
mlx4_0
Infiniband interface OK
Generate the Slurm topology.conf file for Infiniband switches:
# IB switch no. 1: MF0;mell02:IS5030/U1
SwitchName=ibsw1 Nodes=i[001-028]
# IB switch no. 2: MF0;mell03:IS5030/U1
SwitchName=ibsw2 Nodes=finbul,i[029-051],niflfs[1-2],niflopt1
# IB switch no. 3: MF0;mell01:SX6036/U1
SwitchName=ibsw3 Nodes=g[081-100,102-112],h[001-002]
# IB switch no. 4: MF0;mell04:SX6036/U1
# NOTICE: This switch has no attached nodes (empty hostlist)
SwitchName=ibsw4 Nodes=""
# IB switch no. 5: Mellanox 4036 # volt01
SwitchName=ibsw5 Nodes=g[005-008,013-014,016,021-024,029-032,037-040]
# IB switch no. 6: Mellanox 4036 # volt03
SwitchName=ibsw6 Nodes=g[045-048,053-056,061-064,069-072,077-080]
# IB switch no. 7: Mellanox 4036 # volt04
SwitchName=ibsw7 Nodes=g[041-044,049-052,057-060,065-068,073-076]
# IB switch no. 8: Mellanox 4036 # volt02
SwitchName=ibsw8 Nodes=g[001-004,009-012,017-020,025-028,033-036]
# Merging all switches in a top-level spine switch
SwitchName=spineswitch Switches=ibsw[1-8]
It would be great if other sites could test this tool on their Infiniband
network and report bugs or suggest improvements.
--
Ole Holm Nielsen
Department of Physics, Technical University of Denmark
--
Ole Holm Nielsen
PhD, Manager of IT services
Department of Physics, Technical University of Denmark,
Building 307, DK-2800 Kongens Lyngby, Denmark
E-mail: [email protected]
Homepage: http://dcwww.fysik.dtu.dk/~ohnielse/
Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620
