Hi Jonathan,
Do you know what the top level function is or communication pattern? Is
it some type of collective or a pattern that has a many to one. What
might be happening is that since OMPI uses a lazy connections by default
if all processes are trying to establish communications to the same
process you might run into the below.
You might want to see if setting "--mca mpi_preconnect_all 1" helps any.
But beware this will cause your startup to increase. However, this might
give us insight as to whether the problem is flooding a single rank with
connect requests.
--td
Jonathan Dursi wrote:
Hi:
We've got OpenMPI 1.4.1 and Intel MPI running on our 3000 node system. We
like OpenMPI for large jobs, because the startup time is much faster (and
startup is more reliable) than the current defaults with IntelMPI; but we're
having some pretty serious problems when the jobs are actually running. When
running medium- to large- sized jobs (say, anything over 500 cores) over
ethernet using OpenMPI, several of our users, using a variety of very different
sorts of codes, report errors like this:
[gpc-f102n010][[30331,1],212][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection timed out (110)
which sometimes hang the job, or sometimes kill it outright:
[gpc-f114n073][[23186,1],109][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection timed out (110)
[gpc-f114n075][[23186,1],125][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
mpirun: killing job...
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 9513 on node gpc-f123n025
exited on signal 0 (Unknown signal 0).
--------------------------------------------------------------------------
We don't see this problem when the same users, using the same codes, use IntelMPI.
Unfortunately, this only happens intermittently, and only with large jobs, so it
is hard to track down. It seems to happen more reliably with larger numbers of
processors, but I don't know if that tells us something real about the issue, or
just that larger N -> better statistics. For one users code, it definitely
occurs during an MPI_Wait (this particular code has been run on a wide variety of
machines with a wide variety of MPIs -- which isn't proof of correctness of
course, but everything looks fine), for others it is less clear. I don't know if
it's an OpenMPI issue, or just represents a network issue which Intel's MPI
happens to be more tolerant of with the default set of parameters. It's also
unclear whether or not this issue occurred with earlier OpenMPI versions.
Where should I start looking to find out what is going on? Are there
parameters that can be adjusted to play with timeouts to see if the issue can
be localized, or worked around?
- Jonathan
--
Oracle
Terry D. Dontje | Principal Software Engineer
Developer Tools Engineering | +1.650.633.7054
Oracle * - Performance Technologies*
95 Network Drive, Burlington, MA 01803
Email terry.don...@oracle.com <mailto:terry.don...@oracle.com>
