[slurm-dev] Re: Announce: Infiniband topology tool "slurmibtopology.sh" version 0.1

[Ole Holm Nielsen]

On 05/08/2017 08:27 PM, Jeffrey Frey wrote:
The primary problem I've had with ib2slurm is that it segfaults.  There's a bug 
in the ibnetdiscover library -- ib2slurm passes a NULL config pointer to the 
ibnd_discover_fabric() which is supposed to be okay according to the 
documentation, but that function actually requires a config structure to be 
passed to it.

I've forked and updated ib2slurm:


        https://github.com/jtfrey/ib2slurm


There doesn't appear to be much movement on the original project, so I haven't 
put in a pull request against my fork.

It's great that you try to revive the outdated ib2slurm project!  The 
Slurm pages on topology.conf should point to your project in stead of 
the seemingly dead ib2slurm.

That said, the concept of ib2slurm seems to be that you use 
ibnetdiscover to generate a cache file which is subsequently parsed by 
ib2slurm.  My slurmibtopology.sh script uses the same idea: Parse the 
output from ibnetdiscover and generate a topology.conf file.  For me 
personally it was easier to use awk than C for this task because awk 
supports associative arrays.

/Ole


On May 8, 2017, at 10:55 AM, Damien François <damien.franc...@uclouvain.be> 
wrote:

Hi

many thanks for the tools, it works flawlessly here.

I just patched to send the output that does not belong to the topology.conf to 
stderr so I could simply redirect to topology.conf in an automated Slurm 
install procedure

16c16
< echo Verify the Infiniband interface: >&2
---
echo Verify the Infiniband interface:
18c18
< if $IBSTAT -l >&2
---
if $IBSTAT -l
20c20
<    echo Infiniband interface OK >&2
---
        echo Infiniband interface OK
22c22
<    echo Infiniband interface NOT OK >&2
---
        echo Infiniband interface NOT OK
26c26
< cat <<EOF >&2
---
cat <<EOF

Sincerely

damien

On 07 May 2017, at 14:29, Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk> wrote:


I'm announcing an initial version 0.1 of an Infiniband topology tool 
"slurmibtopology.sh" for Slurm.

I had to create a Slurm topology.conf file and needed an automated way to get 
the correct node and switch Infiniband connectivity.  The manual page 
https://slurm.schedmd.com/topology.conf.html refers to an outdated tool 
ib2slurm.

Inspired by the script in 
https://unix.stackexchange.com/questions/255472/text-processing-building-a-slurm-topology-conf-file-from-ibnetdiscover-output
 I decided to write a simpler and more understandable tool.  It parses the output of the 
OFED command "ibnetdiscover" and generates an initial topology.conf file (which 
you may want to edit for readability).

Download the script https://ftp.fysik.dtu.dk/Slurm/slurmibtopology.sh

Example: Running this script on our Infiniband network:

# slurmibtopology.sh
Verify the Infiniband interface:
mlx4_0
Infiniband interface OK

Generate the Slurm topology.conf file for Infiniband switches:

# IB switch no. 1: MF0;mell02:IS5030/U1
SwitchName=ibsw1 Nodes=i[001-028]
# IB switch no. 2: MF0;mell03:IS5030/U1
SwitchName=ibsw2 Nodes=finbul,i[029-051],niflfs[1-2],niflopt1
# IB switch no. 3: MF0;mell01:SX6036/U1
SwitchName=ibsw3 Nodes=g[081-100,102-112],h[001-002]
# IB switch no. 4: MF0;mell04:SX6036/U1
# NOTICE: This switch has no attached nodes (empty hostlist)
SwitchName=ibsw4 Nodes=""
# IB switch no. 5: Mellanox 4036 # volt01
SwitchName=ibsw5 Nodes=g[005-008,013-014,016,021-024,029-032,037-040]
# IB switch no. 6: Mellanox 4036 # volt03
SwitchName=ibsw6 Nodes=g[045-048,053-056,061-064,069-072,077-080]
# IB switch no. 7: Mellanox 4036 # volt04
SwitchName=ibsw7 Nodes=g[041-044,049-052,057-060,065-068,073-076]
# IB switch no. 8: Mellanox 4036 # volt02
SwitchName=ibsw8 Nodes=g[001-004,009-012,017-020,025-028,033-036]
# Merging all switches in a top-level spine switch
SwitchName=spineswitch Switches=ibsw[1-8]

It would be great if other sites could test this tool on their Infiniband 
network and report bugs or suggest improvements.

--
Ole Holm Nielsen
Department of Physics, Technical University of Denmark