Hi Vincent, On Mon, Nov 09, 2009 at 11:45:29AM +0100, Vincent Loechner wrote: > > Martin, > > > I expect problems with sizes larger than 2^31-1, but these array sizes > > are still much smaller. > No, they are bigger, you allocate two arrays of 320 Mdouble : > 2 * 320M * 8 = 5GB. > > Are your processes limited to 4GB virtual memory ?
No, they aren't. There are 16GB on each node plus swap space; I have run processes much bigger than this - in fact this program doesn't even come close to the memory limits on each node. The reason why there is a problem for counts larger than 2^31-1 is that the "count" variable in the MPI library is a MPI_INT - a 32bit integer. BTW: the MPI_Allreduce version fails (i.e., hangs forever) for an array size larger than 176763240. This is not a smooth transition: for 176763240 the program finishes in about 3s, for 176763241 it just hangs. This looks to me as if there is an internal buffer or some other limit that I cannot exceed. Is there such a limit? Or: can somebody confirm that they can run the program with an array size larger than 176763240? Cheers, Martin
