Thanks all. It is now cleared.
Regards, Mahmood ________________________________ From: Damien <dam...@khubla.com> To: Open MPI Users <us...@open-mpi.org> Sent: Saturday, October 27, 2012 7:25 PM Subject: Re: [OMPI users] openmpi shared memory feature Mahmood, To build on what Jeff said, here's a short summary of how diskless clusters work: A diskless node gets its operating system through a physical network (say gig-E), including the HPC applications and the MPI runtimes, from a master server. That master server isn't the MPI head node, it's a separate OS/Network boot server. That's completely separate from how the MPI applications run. The MPI-based HPC applications on the nodes communicate through a dedicated, faster physical network (say Infiniband). There's two separate networks, one for starting and running nodes and one for doing HPC work. On the same node, MPI processes use shared-memory to communicate, regardless of whether it's diskless or not, it's just part of MPI. Between nodes, MPI processes use that faster, dedicated network, and that's regardless of whether it's diskless or not, it's just part of MPI. The networks are separate because it's more efficient. Damien On 27/10/2012 11:00 AM, Jeff Squyres wrote: > On Oct 27, 2012, at 12:47 PM, Mahmood Naderan wrote: > >>> Because communicating through shared memory when sending messages between >>> processes on the same server is far faster than going through a network >>> stack. >> I see... But that is not good for diskless clusters. Am I right? assume >>processes are on a node (which has no disk). In this case, their >>communication go though network (from computing node to server) then IO and >>then network again (from server to computing node). > I don't quite understand what you're saying -- what exactly is your > distinction between "server" and "computing node"? > > For the purposes of my reply, I use the word "server" to mean "one > computational server, possibly containing multiple processors, a bunch of > RAM, and possibly one or more disks." For example, a 1U "pizza box" style > rack enclosure containing the guts of a typical x86-based system. > > You seem to be relating two orthogonal things: whether a server has a disk > and how MPI messages flow from one process to another. > > When using shared memory, the message starts in one process, gets copied to > shared memory, then then gets copied to the other process. If you use the > knem Linux kernel module, we can avoid shared memory in some cases and copy > the message directly from one process' memory to the other. > > It's irrelevant as to whether there is a disk or not. > _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
