Miguel Figueiredo Mascarenhas Sousa Filipe wrote: > H >>> the MPI model assumes you don't have a "shared memory" system.. >>> therefore it is "message passing" oriented, and not designed to >>> perform optimally on shared memory systems (like SMPs, or numa-CCs). >>> >> For many programs with both MPI and shared memory implementations, the >> MPI version runs faster on SMPs and numa-CCs. Why? See the previous >> paragraph... >> > > Of course it does..its faster to copy data in main memory than it is > to do it thought any kind of network interface. You can optimize you > message passing implementation to a couple of memory to memory copies > when ranks are on the same node. In the worst case, even if using > local IP addresses to communicate between peers/ranks (in the same > node), the operating system doesn't even touch the interface.. it > will just copy data from a tcp sender buffer to a tcp receiver > buffer.. in the end - that's always faster than going through a > phisical network link. > > There are a lot of papers about the relative merits of a mixed shared memory and MPI model - OpenMP on-node and MPI inter-node, for example. Generally they seem to show that MPI is at least as good.
It isn't true that copying data in main memory (with a CPU) is always faster than the NIC. Several effects arise: when you copy between address spaces, then the OS overhead of making a shared memory region can be high, the CPUs may not be as good at copying as the NIC, having the CPU do it can trash the caches, and leave the copied data in the wrong cache altogether, and having the CPU do it loses the parallelism you can get by letting a separate processor (the NIC) do the work. -Larry
