Hi Ole,
I tested the script in our topology and I got duplicate entries for nodes
in each switch:
[NVBCABEZA1][root@nvbcabeza1 ~]# ./test.sh |grep -v "#"
Verify the Infiniband interface:
mlx4_0
Infiniband interface OK
Generate the Slurm topology.conf file for Infiniband switches:
SwitchName=ibsw1 Nodes=nvb[14-26]
SwitchName=ibsw2 Nodes=nvb[1-13]
SwitchName=ibsw3
Nodes=nva[28,28-29,29-30,30-31,31-32,32-33,33-34,34-35,35-36,36]
SwitchName=ibsw4
Nodes=nva[37,37-38,38-39,39-40,40-41,41-42,42-43,43-44,44-45,45]
SwitchName=ibsw5
Nodes=nva[46,46-47,47-48,48-49,49-50,50-51,51-52,52-53,53-54,54]
SwitchName=ibsw6
Nodes=nva[55,55-56,56-57,57-58,58-59,59-60,60-61,61],nvblogin[1,1-2,2]
SwitchName=ibsw7 Nodes=nva[1,1-2,2-3,3-4,4,6,6-7,7-8,8-9,9]
SwitchName=ibsw8
Nodes=nva[10,10-11,11-12,12-13,13-14,14-15,15-16,16-17,17-18,18]
SwitchName=ibsw9
Nodes=nva[19,19-20,20-21,21-22,22-23,23-24,24-25,25-26,26-27,27]
SwitchName=ibsw10 Nodes=""
SwitchName=ibsw11 Nodes=""
SwitchName=ibsw12 Nodes=""
SwitchName=ibsw13 Nodes=""
SwitchName=ibsw14 Nodes=""
SwitchName=ibsw15 Nodes=""
SwitchName=ibsw16 Nodes=nvb[27-39],nvbcabeza1
SwitchName=spineswitch Switches=ibsw[1-16]
As you can see nodes appear twice in each switch. Can it be because they
have 2 ib interfaces?
[root@nva10 ~]# ibstat
CA 'mlx4_0'
CA type: MT26428
Number of ports: 1
Firmware version: 2.9.1000
Hardware version: b0
Node GUID: 0x0800380001367b2e
System image GUID: 0x0800380001367b30
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 185
LMC: 0
SM lid: 1
Capability mask: 0x02510868
Port GUID: 0x0800380001367b2f
Link layer: InfiniBand
CA 'mlx4_1'
CA type: MT26428
Number of ports: 1
Firmware version: 2.9.1000
Hardware version: b0
Node GUID: 0x0800380001367b31
System image GUID: 0x0800380001367b33
Port 1:
State: Active
Physical state: LinkUp
Rate: 40
Base lid: 186
LMC: 0
SM lid: 1
Capability mask: 0x02510868
Port GUID: 0x0800380001367b32
Link layer: InfiniBand
Where my topology should be (don't take into account the names of the
switches):
[NVBCABEZA1][root@nvbcabeza1 ~]# cat /opt/perf/etc/slurm/topology.conf
SwitchName=cmc1 Nodes=nva[1-9]
SwitchName=cmc2 Nodes=nva[10-18]
SwitchName=cmc3 Nodes=nva[19-27]
SwitchName=cmc4 Nodes=nva[28-36]
SwitchName=cmc5 Nodes=nva[37-45]
SwitchName=cmc6 Nodes=nva[46-54]
SwitchName=cmc7 Nodes=nva[55-61]
SwitchName=ibswfdr Nodes=nvb[1-39]
SwitchName=troncal Switches=cmc[1-7],ibswfdr
Tell me if you need some more info.
*--Felip Moll Marquès*
Computer Science Engineer
E-Mail - [email protected]
WebPage - http://lipix.ciutadella.es
2017-05-07 14:29 GMT+02:00 Ole Holm Nielsen <[email protected]>:
>
> I'm announcing an initial version 0.1 of an Infiniband topology tool
> "slurmibtopology.sh" for Slurm.
>
> I had to create a Slurm topology.conf file and needed an automated way to
> get the correct node and switch Infiniband connectivity. The manual page
> https://slurm.schedmd.com/topology.conf.html refers to an outdated tool
> ib2slurm.
>
> Inspired by the script in https://unix.stackexchange.com
> /questions/255472/text-processing-building-a-slurm-topology-
> conf-file-from-ibnetdiscover-output I decided to write a simpler and more
> understandable tool. It parses the output of the OFED command
> "ibnetdiscover" and generates an initial topology.conf file (which you may
> want to edit for readability).
>
> Download the script https://ftp.fysik.dtu.dk/Slurm/slurmibtopology.sh
>
> Example: Running this script on our Infiniband network:
>
> # slurmibtopology.sh
> Verify the Infiniband interface:
> mlx4_0
> Infiniband interface OK
>
> Generate the Slurm topology.conf file for Infiniband switches:
>
> # IB switch no. 1: MF0;mell02:IS5030/U1
> SwitchName=ibsw1 Nodes=i[001-028]
> # IB switch no. 2: MF0;mell03:IS5030/U1
> SwitchName=ibsw2 Nodes=finbul,i[029-051],niflfs[1-2],niflopt1
> # IB switch no. 3: MF0;mell01:SX6036/U1
> SwitchName=ibsw3 Nodes=g[081-100,102-112],h[001-002]
> # IB switch no. 4: MF0;mell04:SX6036/U1
> # NOTICE: This switch has no attached nodes (empty hostlist)
> SwitchName=ibsw4 Nodes=""
> # IB switch no. 5: Mellanox 4036 # volt01
> SwitchName=ibsw5 Nodes=g[005-008,013-014,016,021-024,029-032,037-040]
> # IB switch no. 6: Mellanox 4036 # volt03
> SwitchName=ibsw6 Nodes=g[045-048,053-056,061-064,069-072,077-080]
> # IB switch no. 7: Mellanox 4036 # volt04
> SwitchName=ibsw7 Nodes=g[041-044,049-052,057-060,065-068,073-076]
> # IB switch no. 8: Mellanox 4036 # volt02
> SwitchName=ibsw8 Nodes=g[001-004,009-012,017-020,025-028,033-036]
> # Merging all switches in a top-level spine switch
> SwitchName=spineswitch Switches=ibsw[1-8]
>
> It would be great if other sites could test this tool on their Infiniband
> network and report bugs or suggest improvements.
>
> --
> Ole Holm Nielsen
> Department of Physics, Technical University of Denmark
>
