Mahmood,
To build on what Jeff said, here's a short summary of how diskless
clusters work:
A diskless node gets its operating system through a physical network
(say gig-E), including the HPC applications and the MPI runtimes, from a
master server. That master server isn't the MPI head node, it's a
separate OS/Network boot server. That's completely separate from how
the MPI applications run. The MPI-based HPC applications on the nodes
communicate through a dedicated, faster physical network (say
Infiniband). There's two separate networks, one for starting and
running nodes and one for doing HPC work. On the same node, MPI
processes use shared-memory to communicate, regardless of whether it's
diskless or not, it's just part of MPI. Between nodes, MPI processes
use that faster, dedicated network, and that's regardless of whether
it's diskless or not, it's just part of MPI. The networks are separate
because it's more efficient.
Damien
On 27/10/2012 11:00 AM, Jeff Squyres wrote:
On Oct 27, 2012, at 12:47 PM, Mahmood Naderan wrote:
Because communicating through shared memory when sending messages between
processes on the same server is far faster than going through a network stack.
I see... But that is not good for diskless clusters. Am I right? assume processes are on a node (which has no disk). In this case, their communication go though network (from computing node to server) then IO and then network again (from server to computing node).
I don't quite understand what you're saying -- what exactly is your distinction between
"server" and "computing node"?
For the purposes of my reply, I use the word "server" to mean "one computational server,
possibly containing multiple processors, a bunch of RAM, and possibly one or more disks." For example,
a 1U "pizza box" style rack enclosure containing the guts of a typical x86-based system.
You seem to be relating two orthogonal things: whether a server has a disk and
how MPI messages flow from one process to another.
When using shared memory, the message starts in one process, gets copied to
shared memory, then then gets copied to the other process. If you use the knem
Linux kernel module, we can avoid shared memory in some cases and copy the
message directly from one process' memory to the other.
It's irrelevant as to whether there is a disk or not.
