[OMPI users] Using Multiple Gigabit Ethernet Interface

[Jayabrata Chakrabarty]

Hi I have been looking for information on how to use multiple Gigabit Ethernet Interface for MPI communication. 
So far what i have found out is i have to use mca_btl_tcp. 
But what i wish to know, is what IP Address to assign to each Network Interface. I also wish to know if there will be any change in the format of "hostfile"
I have two Gigabit Ethernet Interface on a cluster of 5 nodes at present.
 Thanks in advance

Re: [OMPI users] Using Multiple Gigabit Ethernet Interface

[Jayabrata Chakrabarty]

Thanks Brian, Thanks Michael
I wanted to benchmark the communcation throughput and latency using multiple using gigabit eithernet controller.
So here are the results which i want share with you all
I used .
OpenMPI version 1.0.2a10r9275
Hpcbench 
Two Dell Precision 650 workstation.
The Dell Precision 650 workstation has three separate PCI bus segments.
Segment 1 -> PCI Slot1,2 -> 32 bit, 33MHz, Shared with integrated 1394 
Segment 2 -> PCI SLot3,4 -> 64 bit, 100MHz, Shared with the Gb Ethernet connection 
Segment 3 -> PCI Slot 5 -> Shared with Integrated Ultra 320 controller 
The workstation has Integrated PCI-X 64-bit Intel 10/100/1000 Gigabit Ethernet.
I added three D-Link DGE-530T 1000 Mbps Ethernet Card in Slot2, Slot4 and Slot5 respectively.
As i expected, the Card in slot5 performed better than the cards in other slots. Hereare the results.
(Using Slot2)-# MPI communication latency (roundtrip time) test -- Wed Mar 15 09:19:10 2006# Hosts: DELL <> DELL2# Blocking Communication (MPI_Send/MPI_Recv)# Message size (Bytes) : 40960# Iteration: 7# Test time (Seconds): 0.20
# RTT-time #   Microseconds1    25953.5652    25569.4393    22392.0004    20876.5785    21327.1216    19597.1567    21264.0088    24109.5689    23877.85910   24064.575
# MPI RTT min/avg/max = 19597.156/22903.187/25953.565 usec
--
# MPI communication test -- Wed Mar 15 10:16:22 2006# Test mode: Fixed-size stream (unidirectional) test# Hosts: DELL <> DELL2# Blocking communication (MPI_Send/MPI_Recv)# Total data size of each test (Bytes): 524288000# Message size (Bytes): 104857600# Iteration : 5# Test time: 5.00# Test repetition: 10##  Overall    Master-node  M-process  M-process   Slave-node   S-process  S-process#    Throughput  Elapsed-time  User-mode   Sys-mode  Elapsed-time  User-mode   Sys-mode#    Mbps    Seconds Seconds    Seconds Seconds 
Seconds Seconds1 521.9423 8.04    1.42   6.62    8.04    0.93    7.102 551.5377 7.60    1.20   6.41    7.60    0.77    6.873 552.5600 7.59    1.27   6.32    
7.59    0.82    6.814 552.6328 7.59    1.28   6.31    7.59    0.80    6.835 552.6334 7.59    1.24   6.35    7.59    0.86    6.776 552.7048 7.59    
1.26   6.33    7.59    0.77    6.867 563.6736 7.44    1.22   6.22    7.44    0.78    6.708 552.2710 7.59    1.22   6.37    7.59    0.83    6.809 
520.9938 8.05    1.37   6.68    8.05    0.93    7.1610    535.0131 7.84    1.36   6.48    7.84    0.84    7.04
==
(Using Slot3)-# MPI communication latency (roundtrip time) test -- Thu Mar 16 10:15:58 2006# Hosts: DELL <> DELL2# Blocking Communication (MPI_Send/MPI_Recv)# Message size (Bytes) : 40960# Iteration: 10# Test time (Seconds): 0.20
# RTT-time #   Microseconds1    20094.2042    14773.5123    14846.0154    17756.8205    18419.2906    23394.7997    21840.5968    17727.4949    21822.09510   17659.688
# MPI RTT min/avg/max = 14773.512/18833.451/23394.799 usec
--
# MPI communication test -- Wed Mar 15 09:17:54 2006# Test mode: Fixed-size stream (unidirectional) test# Hosts: DELL <> DELL2# Blocking communication (MPI_Send/MPI_Recv)# Total data size of each test (Bytes): 524288000# Message size (Bytes): 104857600# Iteration : 5# Test time: 5.00# Test repetition: 10##  Overall    Master-node  M-process  M-process   Slave-node   S-process  S-process#    Throughput  Elapsed-time  User-mode   Sys-mode  Elapsed-time  User-mode   Sys-mode#    Mbps    Seconds Seconds    Seconds Seconds 
Seconds Seconds1 794.9650 5.28    1.04   4.24    5.28    0.47    4.812 838.1621 5.00    0.91   4.09    5.00    0.39    4.653 898.3811 4.67    0.84   3.82    
4.67    0.34    4.374 798.9575 5.25    1.03   4.22    5.25    0.40    4.895 829.7181 5.06    0.94   4.11    5.05    0.40    4.696 881.5526 4.76    
0.86   3.90    4.76    0.28    4.527 827.9215 5.07    0.96   4.11    5.07    0.41    4.708 845.6428 4.96    0.87   4.09    4.96    0.38    4.629 
845.6903 4.96    0.90   4.06    4.96 

Re: [OMPI users] Two Gigabit NiCs

[Jayabrata Chakrabarty]

Hi George & Allan
 
First of all i would like to thanks George, for the information. I think with this i can further imporve the performance of my cluster. I was wondering if there is any manual available where all these parameters are listed out. So then i can experiment with them.
 
Allan i would like to bring to your notice the benchmark results i posted here http://www.open-mpi.org/community/lists/users/2006/03/0875.php
 
Maybe this could answer your query as to why you were not getting improved performance.
 
Thanks
Jayabrata

Re: [OMPI users] No increase in performance found using two Gigabit Nics

[Jayabrata Chakrabarty]

Hi Allan, i ran hpbench on a cluster of two machines having just convetional PCI bus and i didnt find any performance improvement. On the contrary the performace degraded. I figured out it may be because conventional PCI bus (33 MHz, 32 bits) provides a bandwidth with 33x32 = 1056 Mbps, hence just one 1000 Mpbs NIC Card is enough to saturate the bandwidth.
 
If you need enhanced performance, you need to get PCI-X (100/133, 62 bits) based machines.
 
thanks 
Jayabrata