We a performing a comparison of HPMPI versus OpenMPI using
Infiniband and
seeing a performance hit in the vicinity of 60% (OpenMPI is slower)
on
controlled benchmarks. Since everything else is similar, we
suspect a
problem with the way we are using or have installed OpenMPI.
Please find attached the following info as requested from
http://www.open-mpi.org/community/help/
<http://www.open-mpi.org/community/help/>
Application: in house CFD solver using both point-point and
collective
operations. Also, for historical reasons it makes extensive use of
BSEND.
We recognize that BSEND's can be inefficient but it is not
practical to
change them at this time. We are trying to understand why the
performance
is so significantly different from HPMPI. The application is mixed
FORTRAN 90 and C built with Portland Group compilers.
HPMPI Version info:
mpirun: HP MPI 02.02.05.00 Linux x86-64
major version 202 minor version 5
OpenMPI Version info:
mpirun (Open MPI) 1.2.4
Report bugs to http://www.open-mpi.org/community/help/
<http://www.open-mpi.org/community/help/>
Configuration info :
The benchmark was a 4-processor job run on a single dual-socket
dual core
HP DL140G3 (Woodcrest 3.0) with 4 GB of memory. Each rank requires
approximately 250MB of memory.
1) Output from ompi_info --all
See attached file ompi_info_output.txt
<< File: ompi_info_output.txt >>
Below is the output requested in the FAQ section:
In order for us to help you, it is most helpful if you can run a
few steps
before sending an e-mail to both perform some basic troubleshooting
and
provide us with enough information about your environment to help
you.
Please include answers to the following questions in your e-mail:
1. Which OpenFabrics version are you running? Please specify where
you
got the software from (e.g., from the OpenFabrics community web
site, from
a vendor, or it was already included in your Linux distribution).
We obtained the software from www.openfabrics.org <www.openfabrics.org
>
Output from ofed_info command:
OFED-1.1
openib-1.1 (REV=9905)
# User space
https://openib.org/svn/gen2/branches/1.1/src/userspace
<https://openib.org/svn/gen2/branches/1.1/src/userspace>
Git:
ref: refs/heads/ofed_1_1
commit a083ec1174cb4b5a5052ef5de9a8175df82e864a
# MPI
mpi_osu-0.9.7-mlx2.2.0.tgz
openmpi-1.1.1-1.src.rpm
mpitests-2.0-0.src.rpm
2. What distro and version of Linux are you running? What is your
kernel version?
Linux xxxxxxxx 2.6.9-64.EL.IT133935.jbtest.1smp #1 SMP Fri Oct 19
11:28:12
EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
3. Which subnet manager are you running? (e.g., OpenSM, a
vendor-specific subnet manager, etc.)
We believe this to be HP or Voltaire but we are not certain how to
determine this.
4. What is the output of the ibv_devinfo command on a known "good"
node
and a known "bad" node? (NOTE: there must be at least one port
listed as
"PORT_ACTIVE" for Open MPI to work. If there is not at least one
PORT_ACTIVE port, something is wrong with your OpenFabrics
environment and
Open MPI will not be able to run).
hca_id: mthca0
fw_ver: 1.2.0
node_guid: 001a:4bff:ff0b:5f9c
sys_image_guid: 001a:4bff:ff0b:5f9f
vendor_id: 0x08f1
vendor_part_id: 25204
hw_ver: 0xA0
board_id: VLT0030010001
phys_port_cnt: 1
port: 1
state: PORT_ACTIVE (4)
max_mtu: 2048 (4)
active_mtu: 2048 (4)
sm_lid: 1
port_lid: 161
port_lmc: 0x00
5. What is the output of the ifconfig command on a known "good" node
and a known "bad" node? (mainly relevant for IPoIB installations)
Note
that some Linux distributions do not put ifconfig in the default
path for
normal users; look for it in /sbin/ifconfig or /usr/sbin/ifconfig.
eth0 Link encap:Ethernet HWaddr 00:XX:XX:XX:XX:XX
inet addr:X.Y.Z.Q Bcast:X.Y.Z.255 Mask:255.255.255.0
inet6 addr: X::X:X:X:X/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1021733054 errors:0 dropped:10717 overruns:0
frame:0
TX packets:1047320834 errors:0 dropped:0 overruns:0
carrier:0
collisions:0 txqueuelen:1000
RX bytes:1035986839096 (964.8 GiB) TX bytes:1068055599116
(994.7 GiB)
Interrupt:169
ib0 Link encap:UNSPEC HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:A.B.C.D Bcast:A.B.C.255 Mask:255.255.255.0
inet6 addr: X::X:X:X:X/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1
RX packets:137021 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:128
RX bytes:12570947 (11.9 MiB) TX bytes:1504 (1.4 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1498664 errors:0 dropped:0 overruns:0 frame:0
TX packets:1498664 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:1190810468 (1.1 GiB) TX bytes:1190810468 (1.1 GiB)
6. If running under Bourne shells, what is the output of the "ulimit
-l" command?
If running under C shells, what is the output of the "limit | grep
memorylocked" command?
(NOTE: If the value is not "unlimited", this FAQ entry
<http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-
pages> and
this FAQ entry
<http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages-more
>
).
memorylocked 3500000 kbytes
Gather up this information and see this page
<http://www.open-mpi.org/community/help/> about how to submit a help
request to the user's mailing list.
