FYI, recently, I was tracking down the source of page faults in our application that has real-time requirements. I found that disabling the sm component (--mca btl ^sm) eliminated many page faults I was seeing. I now have much better deterministic performance in that I no longer see outlier measurements (jobs that usually take 3 ms would sometimes take 15 ms). I did not notice a performance penalty using a network stack.
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Mahmood Naderan Sent: Saturday, October 27, 2012 12:47 PM To: Jeff Squyres Cc: us...@open-mpi.org Subject: EXTERNAL: Re: [OMPI users] openmpi shared memory feature >Because communicating through shared memory when sending messages between >processes on the same server is far faster than going through a network stack. I see... But that is not good for diskless clusters. Am I right? assume processes are on a node (which has no disk). In this case, their communication go though network (from computing node to server) then IO and then network again (from server to computing node). Regards, Mahmood ________________________________ From: Jeff Squyres <jsquy...@cisco.com<mailto:jsquy...@cisco.com>> To: Mahmood Naderan <nt_mahm...@yahoo.com<mailto:nt_mahm...@yahoo.com>>; Open MPI Users <us...@open-mpi.org<mailto:us...@open-mpi.org>> Sent: Saturday, October 27, 2012 6:19 PM Subject: Re: [OMPI users] openmpi shared memory feature On Oct 27, 2012, at 10:49 AM, Mahmood Naderan wrote: > Why openmpi uses shared memory model? Because communicating through shared memory when sending messages between processes on the same server is far faster than going through a network stack. > this can be disabled though by setting "--mca ^sm". > It seems that by default openmpi uses such feature (shared memory backing > files) which is strange. > > Regards, > Mahmood > _______________________________________________ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com<mailto:jsquy...@cisco.com> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
