Well, I figured out how to suppress the readv/writev problems in openmpi -- run it under ktrace! I gave up after the ktrace file reached 46 GB.
This suggests that the "not permitted" failure on writev is a timing problem that appears sporadically. From what I have read about openmpi, a new socket pair is opened for every high level write. If the write tries to transmit before the read socket is established, then the write will fail with this message. So, if some rare event delays the establishment of the read socket, the write fails. This only happens in my case in going from one machine to another, not between processes on the same machine, which kind of makes sense. I have tried three different ethernet cards (realtek, broadcom, and intel) and all do the same thing. The problem maybe isn't even in the ethernet but somewhere higher in the stack, as suggested by Philip Guenther. I am currently writing a tiny openmpi replacement in Go for the small part of mpi that I actually use. I should be able to explore this issue further and maybe even make my model work. Dave Raymond On 12/24/19, Philip Guenther <guent...@gmail.com> wrote: > On Tue, Dec 24, 2019 at 8:14 PM Raymond, David <david.raym...@nmt.edu> > wrote: > >> Openmpi uses readv/writev. I am beginning to think that the timeout >> and permission errors are legit and reflect real conditions. What > > does re do when it receives a write request when it is busy? >> > > 're' does not expose a device, but rather provides network interfaces that > are then used with sockets. What sort of sockets does openmpi use? What > sort of packet loss is generated on this network and what protocols does > openmpi use to recover from that? > > (Lacking both dmesg or kdump, I'll probably have nothing further to > contribute to this thread) > > Philip Guenther > -- David J. Raymond david.raym...@nmt.edu http://physics.nmt.edu/~raymond
