Hi:
We've got OpenMPI 1.4.1 and Intel MPI running on our 3000 node system. We
like OpenMPI for large jobs, because the startup time is much faster (and
startup is more reliable) than the current defaults with IntelMPI; but we're
having some pretty serious problems when the jobs are actually running. When
running medium- to large- sized jobs (say, anything over 500 cores) over
ethernet using OpenMPI, several of our users, using a variety of very different
sorts of codes, report errors like this:
[gpc-f102n010][[30331,1],212][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection timed out (110)
which sometimes hang the job, or sometimes kill it outright:
[gpc-f114n073][[23186,1],109][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection timed out (110)
[gpc-f114n075][[23186,1],125][btl_tcp_frag.c:214:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed: Connection reset by peer (104)
mpirun: killing job...
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 9513 on node gpc-f123n025
exited on signal 0 (Unknown signal 0).
--------------------------------------------------------------------------
We don't see this problem when the same users, using the same codes, use
IntelMPI.
Unfortunately, this only happens intermittently, and only with large jobs, so
it is hard to track down. It seems to happen more reliably with larger
numbers of processors, but I don't know if that tells us something real about
the issue, or just that larger N -> better statistics. For one users code,
it definitely occurs during an MPI_Wait (this particular code has been run on a
wide variety of machines with a wide variety of MPIs -- which isn't proof of
correctness of course, but everything looks fine), for others it is less clear.
I don't know if it's an OpenMPI issue, or just represents a network issue
which Intel's MPI happens to be more tolerant of with the default set of
parameters. It's also unclear whether or not this issue occurred with earlier
OpenMPI versions.
Where should I start looking to find out what is going on? Are there
parameters that can be adjusted to play with timeouts to see if the issue can
be localized, or worked around?
- Jonathan
--
Jonathan Dursi <ljdu...@scinet.utoronto.ca>
