Re: [O-MPI users] [Fwd: Re: The case for an MPI ABI]
On Thu, Mar 10, 2005 at 11:49:52AM -0500, Larry Stewart wrote: > The presentation ignores the issue of instruction set. Even within > the x86 family we have IA-32, EMT 64, and AMD-64. Larry, Thanks for sending some interesting comments. The presentation wasn't intended to be all things to all people. One approach would be to start with only x86 and AMD64/EMT64; that would cover most of the market. I don't think an ABI has to include all processor families to succeed. > Beyond that, we have the situation where toolchains have > incompatible formats and calling standards, even for the same > architecture. Shall we standardize on GCC? On IFC? (I note EKOPATH > is GNU compatible.) On Linux for x86 and AMD64/EM64T, gcc, icc, and pathcc and C++ are all directly compatible. The Fortrans are compatible enough that a single MPI library can deal with all. The calling convention stuff happens to work because MPI doesn't happen to have any calls that hit the "f2c abi" issue. The underscore thing can be handled with multiple symbols for each entry point. The command-line startup thing can be worked around by a clever hack (tm) that I will be happy to share. > Beyond that, an ABI stifles performance. The compiler (in principle > at least) could do interprocedural optimizations between the > application and MPI libraries. Or inlining. I'm not proposing that an ABI be used 100% of the time. And the only commercial compiler publically talking about doing such a thing in production is PathScale, so I think I'd be the first to complain if it actually hurt us -- it doesn't. We expect that ISVs will likely choose the official ABI by default, or the better performance but non-official ABI if they want. No one is stifled. Another important group who isn't stifled is vendors of new interconnect. Today a vendor of a new interconnect can sell easily to anyone who recompiles everything, but anyone who doesn't recompile is hard. Intel, Scali, Scyld, Verari, and HP are all out trying to convince the "we distribute binaries" community that their MPI is the right one to standardize on. A new interconnect vendor will lose in a world where a closed-source MPI is the standard for the "we distribute binary" community. > Even just shipping code as binary forces the vendor into poorly > optimized code, in order to assure functionality on different models > of machines. How much have you talked to ISVs? Most *like* being able to ship a single binary for their application, because they'd rather lose performance on a particular processor sub-model than do more testing. We are encouraged by AMD to improve our EM64T performance so that ISVs can use a single compiler and generate a single binary that's good on both AMD64 and EM64T. AMD even has helped with performance engineering! But there's no need argue this point, it's not really relevant to the MPI ABI issue, and the people who prefer distributing binaries are going to continue to do so no matter if there's an MPI ABI or not. > Use the source. If you'd like to use the source, then please continue to do so. The people who find the idea of an ABI compelling are the people listed in the presentation: ISVs, and open source projects which want to distribute flexible binary RPMs because their typical user doesn't want to recompile. On the flip side, groups such as the OpenMPI project would gain by supporting an ABI because they'd be able to run with applications "built for MPICH" without having to recompile. The OpenMPI folks may not find this compelling; your typical programmer at the national labs doesn't mind recompiling. But if you wanted to do a study of whether or not OpenMPI improved the performance of some ISV code, I assure you that an ABI would make that a lot easier. -- greg
Re: [O-MPI users] [Fwd: Re: The case for an MPI ABI]
On Fri, Mar 11, 2005 at 02:21:58PM +1100, Stuart Midgley wrote: > One major implementation issue is the equivalent of mpirun (which I > assume would be part of an ABI?) - or the startup requirements of > different MPI's. This may or may not be part of an ABI. The reason to not include it is that it's kind of hard. Right now everyone does different stuff. The reason to include it is that it would make integrating with batch systems less of a headache for system administrators. Right now most only support 1 MPI version because this is annoying. > The other issue we are concerned about is that an ABI doesn't resolve > one of the central issues. While you might have different MPI's with > the same ABI, different mpi's behave differently and can cause a code > to behave differently. An ISV would still have to verify their code > against all MPI's they wish to support. For example, an ISV might, by > accident, make an assumption about the small message size and their > code might hang on different MPI's. For exactly this reason, I expect the average ISV to only want to support 1 MPI. However, testing multiple MPIs is far, far easier if you don't have to recompile for each one. So a uniform is a major step in the right direction, and any ISV which already supports more than one MPI is going to love it. And if testing became easier, people might actually fix their code. OK, *now* I'm dreaming... ;-) -- greg
Re: [O-MPI users] Fwd: Thoughts on an MPI ABI
Jeff, One of the interesting aspects of an ABI MPI is that some folks won't be convinced that it's interesting until they've had it proven that it will work, both technically and socially, which is putting the cart before the horse to a certain extent. It's also not helpful that all I distributed were power-point slides; I did have a detailed writeup, but it gave proposed solutions to too many problems, which is likely to cause more arguments than it would solve. In any case, discussion is a good thing, and here are some answers to the issues you raise. I'm afraid that a casual observer won't necessarily learn much from our exchange about whether or not an ABI is possible or desirable. I also don't think I have all the answers, but a few of the issues you raise do delve into things that I've worked on in the past. 1. An ABI still leaves you vulnerable to different C++ and F90 issues. On AMD64/EM64T and x86, Intel, gcc, and PathScale have compatible C++ implementations. This is not a coincidence. The F90 issue is real, but note that it reduces to N libraries for N compilers, instead of N*M libraries for N compilers and M MPIs. 1a. You're vulnerable to OS issues. Yes. People who distribute binaries today already deal somewhat successfully with that. We distribute only 4 sets of compiler RPMs, but they happen to work successfully on RedHat Enterprise, Fedora, and 9, SUSE Professional and Enterprise, Mandrake, Debian, Gentoo, White Hat, Rocks, and a few others I've forgotten because they're too obscure. 2. Choosing an MPI implementation One existing solution is "modules". It takes the guesswork out of setting and resetting PATH. It's definitely something that has to be scripted or users can't use it reliably. You are correct that setting LD_LIBRARY_PATH in an MPI job is an issue, but this needs a fix anyway, as users run up against it all the time -- "Your Fortran docs say I need to set this environment variable to change the output format; it works with a serial code, but doesn't work with MPI! Help!" 3. This isn't an immediately perfect solution for all ISVs Well, yes. I actually said that in my slides. For the average ISV, being able to test other MPIs without recompiling is a first step towards being able to eventually support more than one. But it's not the only step. An above-average ISV will then be willing to let customers use other MPIs, as long as any bugs are replicated using a supported platform (i.e. using OpenMPI over TCP/IP). I expect it to take a while for the average ISV to get there, but at least they have a chance. Today they have no chance. You also mention that some ISVs tend to distribute static linked executables. This is true. Were I an ISV, I would distribute such an executable for debugging purposes. 4. Non-standardized parts of MPI are an issue in real programs. I agree with this one. I think these issues should be discussed as part of the ABI effort. What parts are the worst problems, and can we successfully standardize them? 5. Biologists and Chemists don't care about an ABI. Yes, but they do care about having their apps just work. And an ABI helps that. I can imagine a future in which an rpm for a biology app distributes a test suite, and MPI implementers / interconnect vendors who care will download the rpm and test it. That's easier with an ABI than without; with an ABI you don't have to rebuild (which, for a poorly packaged app, can be very tricky for a 3rd party to do), and you can be fairly sure that you didn't introduce/delete a bug by recompiling. 6. ISVs can ship MPI-independent apps today using a thin abstraction layer. Yes, but that wastes money and complicates testing. I wouldn't be surprised if few of them did that. I suspect the ISVs that did that, if any, would like an ABI more than their current system. 7. It's expensive to change sizes and types of MPI handles to a new standard Look at your f77 interface -- it's a roadmap for converting from fortran integer arguments to pointers. This is what I suggest you do for C/C++. I'd be happy to write the code for you, actually. 8. Embedded MPI might have different constraints An ABI doesn't have to be all things to all people. I find that most HPC embedded systems are based on a fairly general-purpose platform, so an ABI may manage to reach most of today's embedded players. (We do do business in that space.) 9. Is recompiling really that difficult? I don't think you or I are the right people to answer that one, we recompile all the time. But I assure you that the compiler people in my company would answer YES! YES! Why? Because compilers are heavily QAed, and we can name numerous source files in our compiler that have obscure bugs appear if compiled with gcc -O2 instead of -O0. These bugs come and go, too, as we rearrange our source code. So for them, they would never send a compiler binary out the door that had not been fully QAed -- that exact binary. No "let's recompile for one fix and then not
Re: [O-MPI users] [Fwd: Re: The case for an MPI ABI]
On Thu, Mar 17, 2005 at 12:29:22PM +, Neil Storer wrote: > Be careful when you say: Neil, I think that you'll find that pathf90 accepts -1 for TRUE, so this is easily handled by the binding for MPI. I'd have to write some test programs to be sure, and I'll get back to you on that. I think the difference is whether the compiler tests the sign bit or non-zero for TRUE. I'm not sure if either our compiler or PGI's ever does arithmetic on logical vlues. This is definitely a bug in our documentation, though, thanks for pointing that out. -- greg
[O-MPI users] Re: [Beowulf] Alternative to MPI ABI
> Create a new software project (preferably open source, preferably with > an BSD-like license so that ISV's can incorporate this software into > their products) that provides a compatibility layer for all the > different MPI implementations out there. Let's call it MorphMPI. Jeff, A similar idea was actually written up by Bill Gropp and was mentioned by him 5 weeks ago on the beowulf list. Quoth he: | I wrote a paper that appeared in the EuroPVMMPI'02 meeting that discusses | the issues of a common ABI. The paper is "Building Library Components That | Can Use Any MPI Implementation" and is available as | http://www-unix.mcs.anl.gov/~gropp/bib/papers/2002/gmpishort.pdf . I think this overall idea falls short of the benefit of an ABI for a couple of reasons. The first is that it's unlikely to get wide distribution if it doesn't come with MPI implementations. The second is that it's harder to maintain "out of tree"; the minute that an MPI implementation changes something, MorphMPI is going to be broken. You keep on coming back to this: > 2. Cancel the MPI Implementor's Ultimate Prize Fighting Cage Match on > pay-per-view (read: no need for time-consuming, potentially fruitless > attempts to get MPI implementors to agree on anything) Was there a big fight over the Fortran interface? It nails down the types because it has to. All the ABI does is make C look like Fortran; no internals need change in any implementation. -- greg
[O-MPI users] Re: [Beowulf] Alternative to MPI ABI
On Fri, Mar 25, 2005 at 02:59:18PM -0500, Patrick Geoffray wrote: > I don't see it that way. First, the implementations of the translation > layers will be done by each MPI implementations. In which case it's basically the same as doing an ABI. Or did I miss something? Does this somehow save a significant amount of work for anyone? > >Was there a big fight over the Fortran interface? It nails down the > >types because it has to. All the ABI does is make C look like Fortran; > >no internals need change in any implementation. > > You don't change internals, you translate them. Let say you use pointers > in your MPI implementation and the common layer specifies integers. In > your translation layer, you translate pointers into integers by putting > them in a table. You have as much work as your internals are far from > the common interface and, hopefully, it will be a midpoint for everybody. Patrick, if you read what both Jeff and I wrote, I believe it's clear we both understand that part, because we've both mentioned that particular implementation solution. What I was trying to understand was why Jeff thought this was a huge nightmare. -- greg
[O-MPI users] Re: [Beowulf] Alternative to MPI ABI
On Fri, Mar 25, 2005 at 06:03:15PM -0500, Patrick Geoffray wrote: > What Jeff thought is a nightmare, I believe, is to have to decide a > common interface and then force the MPI implementations to adopt this > interface internally instead of having them translating on the fly. Ah. But no one ever suggested that, so we're all set -- it's an artifact of the poor communication content of PowerPoint slides that anyone thought I had suggested altering everyone's internals. > > something? Does this somehow save a significant amount of work for > > anyone? > > It does not, but the pill is much easier to swallow because nobody has > to fight to try to impose the interface they happen to use. Am I still > drunk and missing something big ? If I understand it correctly, MorphMPI imposes the same interface as an ABI -- a common mpi.h. The only thing it avoids is having a shared library with a common name and interface; instead it will have a separate routine per MPI implementation that dlopens all the appropriate libs for the implementation in use, with the usual chaos of trying to find where they are located. In any case, I think this sort of discussion is more of an implementation detail than a fundamental thing that would obviate having an ABI... either way you're going to want to pick the right contents for mpi.h. Apple or Orange, it's the same committee process. -- greg
Re: [O-MPI users] Re: [Beowulf] Alternative to MPI ABI
On Fri, Mar 25, 2005 at 05:49:13PM -0500, Jeff Squyres wrote: > MorphMPI (or, as Patrick suggests, we need a cooler name -- PatrickMPI? > ;-) ) is the work of 1 grad clever student (or anyone else industrious > enough). Elapsed time: a few months. Right. > Making even 2 MPI implementations agree on an ABI is an enormous amount > of work. Given that two major MPI implementations take opposite sides > on the pointers-vs.integers for MPI handles debate (and I suspect that > neither is willing to change), just getting them to agree on one of > them will be a major amount of work. Then changing the internals of > one of those MPIs to match the other is another enormous amount of work > (death by a million cuts). You yourself said how MPI implementers would actually implement this without needing to change any internals: Make the C interface routines do the same thing that F77 does today. Elapsed time: a few months, same as MorphMPI. No internals need to be changed. I suppose the good news is that if this is your main objection, then it's gone. > Also, as I pointed out in my original alternate proposal, with > PatrickMPI, only those who want to use an ABI will use it. Those who > do *not* want an ABI do not have to have it forced upon them. I missed where anyone was being forced to do anything. MPI implementers can support the ABI alongside their current interface or not, it's their choice. -- greg
Re: [O-MPI users] Re: [Beowulf] Alternative to MPI ABI
On Sat, Mar 26, 2005 at 06:47:41AM -0500, Jeff Squyres wrote: > Regardless of which way you choose, your statement "No internals have > to change" is inaccurate. At a minimum, *EVERY* MPI API function in > somebody's implementation will have to change. That's what I call the interface, yes. It's a similar size effort to creating an F77 interface in an MPI implementation. We've agreed on this several times already; I'm not sure why we need to agree on it again, or why the exact definition of the word "internals" is so important. > I guess I don't understand your reluctance to accept a MorphMPI-like > solution: You have repeated your original MorphMPI attributes. I responded to them, and I don't see any sign that you've read my response. This is not the way discussions are usually held. -- greg
Re: [O-MPI users] Re: [Beowulf] Alternative to MPI ABI
On Sun, Apr 03, 2005 at 02:19:39PM -0400, Jeff Squyres wrote: > If so, are we therefore in agreement that a MorphMPI-like approach is a > good first step? No, because we apparently disagree about what MorphMPI is. You claim it's a lot less work than an ABI; I claim it's about the same. We both agreed that in both cases, most of the work was building a C interface which is similar to the existing F77 interface. So where's the extra work you say an ABI needs? > Who, exactly, wants an MPI ABI? If you look at the original presentation, the biggest win is for 1) Customers who run "canned" applications, both commercial and free 2) People who produce these apps 3) Vendors of unusual interconnect hardware. Recall that I also said that MPI implementers have the least motivation, because they have to do some work. For example, since you didn't see any benefit from avoiding recompiling, it's no surprise that you don't see anything but a negative from an ABI. > Have there been any formal surveys conducted? Not yet. We're at the trial balloon phase. Obviously demand has to be proven before anyone is going to do much work on an ABI. In April I will be attending 2 conferences with lots of people in the (1)-(3) categories; that will give me a chance to talk to a bunch of people who tend not to post to email lists. -- greg
Re: [O-MPI users] MIPS64 - The Long-Awaited Config File
On Sat, Sep 10, 2005 at 04:45:48PM -0500, Brian Barrett wrote: > I think that this is a fairly easy fix - Irix identifies any MIPS > chip as a mips-* from config.guess, but Linux apparently makes a > distinction between mips and mips64. That's because there is a difference between mips and mpis64. The assembly syntax looks similar, but there are subtle pitfalls involved in using the same code for both. You can look at the Linux kernel code for atomics in mips and mips64 to see if there are any differences. -- greg, who used to play around with MIPS64 in the good old days
Re: [O-MPI users] MIPS64 - The Long-Awaited Config File
On Sun, Sep 11, 2005 at 09:24:02AM -0500, Brian Barrett wrote: > I'll admit that my only interaction with the MIPS architectures are > the R4K and above in SGI systems, so I'm a bit out of my league. The R4K was the first MIPS64 chip. I bought a second-hand manual for it online for a few bucks. There are some later additions to the architecture, but they probably won't affect this task. -- greg
Re: [O-MPI users] MIPS64 Trek - The story so far
On Mon, Sep 12, 2005 at 07:24:43PM -0500, Brian Barrett wrote: > By the way, to clarify, the assembly > has been tested on a MIPS R14K in 64 bit mode (and 32 bit mode using > SGI's n32 ABI -- it will not work with their o32 ABI). In Linux/binutils lingo, that means it's MIPS64 code, and not MIPS. -- greg
Re: [O-MPI users] [Beowulf] MPI ABI
> ignoring the politics for a moment, what are the technical sticking points? MPI types values of constants Fortran name-mangling Fortran LOGICAL program startup (optional) > for instance, I have the impression that the linux x86_64 ABI is reasonably > well-defined, It mostly works. We have run MPICH-1 based commercial codes by playing shared library tricks. -- greg
Re: [O-MPI users] [Beowulf] MPI ABI
> The government is one of the few forces that could mandate a proper MPI > ABI at this point in time; They certainly aren't the only ones -- vendors of proprietary applications that use MPI plus vendors of interconnect hardware get significant benefits from an ABI. Anyone who wants to distribute binaries of open-source MPI apps would also benefit, but they generally don't get to use the verb "mandate". -- greg
Re: [O-MPI users] [Beowulf] MorphMPI based on fortran itf (was: MPI ABI)
On Wed, Oct 12, 2005 at 12:05:13PM +0100, Ashley Pittman wrote: > Thirdly is the performance issue, any MPI vendor worth his salt tries > very hard to reduce the number of function calls and library's between > the application and the network, adding another one is a step in the > wrong direction. This may not matter so much for ethernet clusters but > certainly for some people the software stack accounts for a surprising > percentage of "network" latency. OK, so that's a new item for the Technical List Of Things To Do: measure the overhead. I suspect it'll turn out to be small, even for interconnects that care about 50 nanoseconds of additional overhead. -- greg
Re: [O-MPI users] [Beowulf] MorphMPI based on fortran itf (was: MPI ABI)
On Wed, Oct 12, 2005 at 07:06:54PM +0100, Ashley Pittman wrote: > As it turns out I'm in a position to measure this fairly easily, our MPI > sits on top of a library called libelan, this does all the tag matching > at a very low level, all MPI does is convert the communicator into a bit > pattern, calculate the length from the type and count and convert from > lrank to grank, passing the call on. As you point out later, that's a lot more than what needs to be measured. > Regardless of the numbers this is a *high performance* industry and > doing this would be a step in the wrong direction. If the hit is tiny and the benefit is large, the high performance industry will gladly adopt it. I have yet to see, for example, anyone rewrite their MPI code from f77 into C in order to avoid the current overhead for the f77 MPI interface. In an ideal world, an MPI ABI will have the same f77 overhead as today, and C overhead similar to f77. In any case, accurately measuring this overhead and then discussing it is better than discussing it without accurately measuring. -- greg
Re: [O-MPI users] pathscale 2.1/2.3 build problem
Dries, We have built OpenMPI-1.0 with our compilers, but don't build SVN. In your case you seem to have some kind of SUSE weirdness going on. Is it possible you have a gcc-3.3 compatibility package of some kind installed? We should probably take this conversation over to supp...@pathscale.com, but the next step is to try "pathcc -v hello.c", and also /usr/bin/gcc -E "foo bar" -- greg
Re: [OMPI users] Toubles with Open-mpi on AMD opteron with gfortran/g95
On Fri, Feb 24, 2006 at 04:44:19PM +0100, Benoit Semelin wrote: > Well I have actually tried with 2 different versions: 8.0.034 and > 8.1.018. The config.log is attached (I hope this works on the mailing > list...) Looks like ifort on your system is generating 32-bit .o files by default. configure:20653: gcc -O3 -DNDEBUG -fno-strict-aliasing -I. -c conftest.c configure:20660: $? = 0 configure:20670: ifort conftestf.f conftest.o -o conftest ld: warning: i386:x86-64 architecture of input file `conftest.o' is incompatible with i386 output Rainer wrote: > >PS: When compiling OpenMPI are You using a combination of gcc for C/C++ > >and ifort for Fortan compilation? > >This will not work, as the compilers have different views on e.g. boolean > >variables and values, so parameter passing from Your code into OpenMPI > >will be botched up. Currently, You may only use C / Fortran Compiler from > >one vendor. That's kind of surprising, I don't know why you would want to require that bool and LOGICAL have the same values. We (and Intel) both do bool like g++ does, but we each do LOGICAL to match the history of our Fortran compilers. I never imagined anyone would think these should be the same. -- greg
Re: [OMPI users] MPI_COMM_SPAWN versus OpenMPI 1.0.1
On Wed, Mar 01, 2006 at 09:56:38AM -0500, George Bosilca wrote: > But, even if it was > right to do so, it should at least print a warning message to inform > me that I was stupid enough to do such basic mistakes ... Most compilers don't because they don't want to print warnings for codes which have sequence numbers -- that's what goes in the last 8 columns on your 80-column Hollerith punch card, you see. A few default to warning and have a switch to turn the warning off. -- greg
Re: [OMPI users] OpenMPI 1.0.x and PGI pgf90
On Thu, Mar 09, 2006 at 09:13:46PM -0500, Brian Barrett wrote: > I think I have this fixed on the trunk. It looks like PGI tried to > make the 6.1 compilers support GCC inline assembly, but it doesn't > look like it's 100% correct, ... and that's no surprise. The spec in the gcc info pages doesn't reflect reality, and with our compiler, I filed 20 bugs before we got gmp (gnu multi-precision library, a heavy user of inline assembly) to work. Doctor, it hurts when I do this... -- greg
Re: [OMPI users] OMPI 1.1 build failure with Intel 9.0
On Tue, Jul 11, 2006 at 12:23:16PM -0400, George Bosilca wrote: > I doubt that icc should know anything about > the gxx_personality. In fact it look like icc is trying to use some > libraries compiled with g++. As an aside, both Intel C++ and PathScale C++ are 100% g++ compatible. Symbols like this are part of the g++ compatibility: [lindahl@eng-23 2.4]$ nm ipl.so| grep gxx 000a21dc V DW.ref.__gxx_personality_v0 U __gxx_personality_v0@@CXXABI_1.2 -- greg
Re: [OMPI users] Does current Intel dual processor support MPI?
On Tue, Sep 05, 2006 at 11:50:54AM -0400, George Bosilca wrote: > Yes and yes. However, these architectures fit better on a different > programming model. If you want to get the max performance out of > them, a OMP approach (instead of MPI) is more suitable. Eh? You mean all of those examples of codes with OMP and MPI versions that run faster with MPI are a figment of my imagination? -- greg
Re: [OMPI users] IB HCA support
On Tue, Oct 03, 2006 at 12:01:37PM -0600, Troy Telford wrote: > I can't claim to know which ones are *known* to work, but I've never seen > an IB HCA that didn't work with Open MPI. Ditto. Ours works fine with the OFED stack, and also there's "accelerated" support for our PSM software interface checked into the Open MPI trunk. -- greg
Re: [OMPI users] efficient memory to memory transfer
On Tue, Nov 07, 2006 at 05:02:54PM +, Miguel Figueiredo Mascarenhas Sousa Filipe wrote: > if your aplication is on one given node, sharing data is better than > copying data. Unless sharing data repeatedly leads you to false sharing and a loss in performance. > the MPI model assumes you don't have a "shared memory" system.. > therefore it is "message passing" oriented, and not designed to > perform optimally on shared memory systems (like SMPs, or numa-CCs). For many programs with both MPI and shared memory implementations, the MPI version runs faster on SMPs and numa-CCs. Why? See the previous paragraph... -- greg
Re: [OMPI users] efficient memory to memory transfer
On Wed, Nov 08, 2006 at 09:57:18AM -0500, Hugh Merz wrote: > The conventional wisdom of pure MPI being as good as hybrid models > is primarily driven by the fact that people haven't had much incentive > to re-write their algorithms to support both models. Actually, I was thinking of apps where people spent lots of money to add hybrid models to their code, and found it didn't win much. But it's hard for reality to catch up with a bad religion. > If current HPC trends continue into the future there is going to be > increased motivation to implement fine-grained parallelism in > addition to MPI. And to make MPI do finger grain parallelism, too... which is why InfiniPath IB HCAs do so well on small messages. -- greg
Re: [OMPI users] efficient memory to memory transfer
On Wed, Nov 08, 2006 at 12:25:20PM +, Miguel Figueiredo Mascarenhas Sousa Filipe wrote: > > Unless sharing data repeatedly leads you to false sharing and a loss > > in performance. > > what does that mean.. I did not understand that. Google indexes a bunch of good webpages on "false sharing". > > For many programs with both MPI and shared memory implementations, the > > MPI version runs faster on SMPs and numa-CCs. Why? See the previous > > paragraph... > > Of course it does..its faster to copy data in main memory than it is > to do it thought any kind of network interface. MPI implementations typically copy data in main memory when it is talking on the same node. If it's faster to use the network interface, and sometimes it is due to DMA engines, then your friendly MPI implementer is likely to be taking advantage of it. > But, for instance.. try to benchmark real applications with a MPI and > posix threads implementations in the same numa-cc or big SMP machine.. > my bet is that posix threads implementation is going to be faster.. And your bet is often wrong. -- greg
Re: [OMPI users] How to set paffinity on a multi-cpu node?
On Fri, Dec 01, 2006 at 11:51:24AM +0100, Peter Kjellstrom wrote: > This might be a bit naive but, if you spawn two procs on a dual core dual > socket system then the linux kernel should automagically schedule them this > way. No, we checked this for OpenMP and MPI, and in both cases wiring the processes to cores was significantly better. The Linux scheduler (still) tends to migrate processes to the wrong core when OS threads and processes wake up and go back to sleep. Just like the OpenMPI guys, we don't have a clever solution for the "what if the user wants to have 2 OpenMP or MPI jobs share the same node?" Well, I have a plan, but it's annoying to implement. -- greg
Re: [OMPI users] How to set paffinity on a multi-cpu node?
On Sat, Dec 02, 2006 at 10:31:30AM -0500, Jeff Squyres wrote: > FWIW, especially on NUMA machines (like AMDs), physical access to > network resources (such as NICs / HCAs) can be much faster on > specific sockets. Yes, the penalty is actually 50 ns per hop, and you pay it on both sides. So our 1.28 usec latency rises to 1.38 if you are on the other socket on both ends, using a 2 socket motherboard. In our MPI we also have code that pays attention to the NUMA topology if you have 2 of our HTX HCAs in a system. -- greg
