Hi, I'm evaluating Open MPI 1.4.2 on one of our BladeCenters and I'm getting via InfiniBand about 1550 MB/s and via shared memory about 1770 for the PingPong benchmark in Intel's MPI benchmark. (That benchmark is just an example, I'm seeing similar numbers for my own codes.)
Each node has two AMD hex-cores and two 40 Gbps InfiniBand ports, so I
wonder if I shouldn't be getting a significantly higher throughput on
InfiniBand. Considering the CPUs' memory bandwidth, I believe that
shared memory throughput should be much higher as well.
Are those numbers what is to be expected? If not: any ideas how to
debug this or tune Open MPI?
Thanks in advance
-Andreas
ps: if it's any help, this is what iblinkinfo is telling me
(tests were run on faui36[bc])
faui36a ~ # iblinkinfo -P 1
Switch 0x0008f10500380136 IBM HSSM:
4 1[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 1 1[ ]
"faui36a HCA-1" ( )
4 2[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 9 1[ ]
"MT25204 InfiniHostLx Mellanox Technologies" ( )
4 3[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 4[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 5[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 6[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 7[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 8[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 9[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 10[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 11[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 12[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 13[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 14[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 15[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 16[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 17[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 6 1[ ]
"faui36b HCA-1" ( )
4 18[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 8 1[ ]
"faui36c HCA-1" ( )
4 19[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 3 1[ ]
"faui36d HCA-1" ( )
4 20[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 2 1[ ]
"faui36e HCA-1" ( )
4 21[ ] ==( 4X 5.0 Gbps Active/ LinkUp)==> 7 1[ ]
"faui36f HCA-1" ( )
4 22[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 23[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 24[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 25[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 26[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 27[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 28[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 29[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 30[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 31[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 32[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
4 33[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 34[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 35[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
4 36[ ] ==( 4X 2.5 Gbps Down/Disabled)==> [ ] "" (
)
faui36a ~ # iblinkinfo -P 2
Switch 0x0005ad0000080ba2 Topspin Switch:
10 1[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 13 2[ ]
"faui36b HCA-1" ( )
10 2[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 15 2[ ]
"faui36c HCA-1" ( )
10 3[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 12 2[ ]
"faui36d HCA-1" ( )
10 4[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 11 2[ ]
"faui36e HCA-1" ( )
10 5[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 14 2[ ]
"faui36f HCA-1" ( )
10 6[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 7[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 8[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 9[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 10[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 11[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 12[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 13[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 14[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 15[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 16[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 17[ ] ==( 4X 2.5 Gbps Active/ LinkUp)==> 5 2[ ]
"faui36a HCA-1" ( )
10 18[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 19[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 20[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 21[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 22[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 23[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
10 24[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" (
)
--
==========================================================
Andreas Schäfer
HPC and Grid Computing
Chair of Computer Science 3
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
+49 9131 85-27910
PGP/GPG key via keyserver
I'm a bright... http://www.the-brights.net
==========================================================
(\___/)
(+'.'+)
(")_(")
This is Bunny. Copy and paste Bunny into your
signature to help him gain world domination!
pgpO4VwtvaiDu.pgp
Description: PGP signature
