Hmmm....I'll see what I can do about the error message. I don't think there is much in 1.6 I can do, but in 1.7 I could generate an appropriate error message as we have a way to check the topologies.
On Dec 20, 2012, at 7:11 AM, Brock Palen <bro...@umich.edu> wrote: > Ralph, > > Thanks for the info, > That said I found the problem, one of the new nodes, had Hyperthreading on, > and the rest didn't so all the nodes didn't match. A quick > > pdsh lstopo | dshbak -c > > Uncovered the one different node. The error just didn't give me a clue to > that being the cause, which was very odd: > > Correct node: > [brockp@nyx0930 ~]$ lstopo > Machine (64GB) > NUMANode L#0 (P#0 32GB) + Socket L#0 + L3 L#0 (20MB) > L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0) > L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#1) > L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2) > L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3) > L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 + PU L#4 (P#4) > L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5) > L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 + PU L#6 (P#6) > L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 + PU L#7 (P#7) > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (20MB) > L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 + PU L#8 (P#8) > L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 + PU L#9 (P#9) > L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 + PU L#10 (P#10) > L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 + PU L#11 (P#11) > L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 + PU L#12 (P#12) > L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 + PU L#13 (P#13) > L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 + PU L#14 (P#14) > L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 + PU L#15 (P#15) > > > Bad node: > [brockp@nyx0936 ~]$ lstopo > Machine (64GB) > NUMANode L#0 (P#0 32GB) + Socket L#0 + L3 L#0 (20MB) > L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 > PU L#0 (P#0) > PU L#1 (P#16) > L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 > PU L#2 (P#1) > PU L#3 (P#17) > L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 > PU L#4 (P#2) > PU L#5 (P#18) > L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 > PU L#6 (P#3) > PU L#7 (P#19) > L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 > PU L#8 (P#4) > PU L#9 (P#20) > L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 > PU L#10 (P#5) > PU L#11 (P#21) > L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 > PU L#12 (P#6) > PU L#13 (P#22) > L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 > PU L#14 (P#7) > PU L#15 (P#23) > NUMANode L#1 (P#1 32GB) + Socket L#1 + L3 L#1 (20MB) > L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 > PU L#16 (P#8) > PU L#17 (P#24) > L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 > PU L#18 (P#9) > PU L#19 (P#25) > L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 > PU L#20 (P#10) > PU L#21 (P#26) > L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 > PU L#22 (P#11) > PU L#23 (P#27) > L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 > PU L#24 (P#12) > PU L#25 (P#28) > L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 > PU L#26 (P#13) > PU L#27 (P#29) > L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 > PU L#28 (P#14) > PU L#29 (P#30) > L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 > PU L#30 (P#15) > PU L#31 (P#31) > > Once I removed that node from the pool the error went away, and using > bind-to-core and cpus-per-rank worked. > > I don't see how an error message of the sort given would ever lead me to find > a node with 'more' cores, even if fake, I was looking for a node that had a > bad socket or wrong part. > > > Brock Palen > www.umich.edu/~brockp > CAEN Advanced Computing > bro...@umich.edu > (734)936-1985 > > > > On Dec 19, 2012, at 9:08 PM, Ralph Castain wrote: > >> I'm afraid these are both known problems in the 1.6.2 release. I believe we >> fixed npersocket in 1.6.3, though you might check to be sure. On the >> large-scale issue, cpus-per-rank well might fail under those conditions. The >> algorithm in the 1.6 series hasn't seen much use, especially at scale. >> >> In fact, cpus-per-rank has somewhat fallen by the wayside recently due to >> apparent lack of interest. I'm restoring it for the 1.7 series over the >> holiday (currently doesn't work in 1.7 or trunk). >> >> >> On Dec 19, 2012, at 4:34 PM, Brock Palen <bro...@umich.edu> wrote: >> >>> Using openmpi 1.6.2 with intel 13.0 though the problem not specific to the >>> compiler. >>> >>> Using two 12 core 2 socket nodes, >>> >>> mpirun -np 4 -npersocket 2 uptime >>> -------------------------------------------------------------------------- >>> Your job has requested a conflicting number of processes for the >>> application: >>> >>> App: uptime >>> number of procs: 4 >>> >>> This is more processes than we can launch under the following >>> additional directives and conditions: >>> >>> number of sockets: 0 >>> npersocket: 2 >>> >>> >>> Any idea why this wouldn't work? >>> >>> Another problem the following does what I expect, two 2 socket 8 core >>> sockets. 16 total cores/node. >>> >>> mpirun -np 8 -npernode 4 -bind-to-core -cpus-per-rank 4 hwloc-bind --get >>> 0x0000000f >>> 0x0000000f >>> 0x000000f0 >>> 0x000000f0 >>> 0x00000f00 >>> 0x00000f00 >>> 0x0000f000 >>> 0x0000f000 >>> >>> But fails at large scale: >>> >>> mpirun -np 276 -npernode 4 -bind-to-core -cpus-per-rank 4 hwloc-bind --get >>> >>> -------------------------------------------------------------------------- >>> An invalid physical processor ID was returned when attempting to bind >>> an MPI process to a unique processor. >>> >>> This usually means that you requested binding to more processors than >>> exist (e.g., trying to bind N MPI processes to M processors, where N > >>> M). Double check that you have enough unique processors for all the >>> MPI processes that you are launching on this host. >>> You job will now abort. >>> -------------------------------------------------------------------------- >>> >>> >>> >>> Brock Palen >>> www.umich.edu/~brockp >>> CAEN Advanced Computing >>> bro...@umich.edu >>> (734)936-1985 >>> >>> >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
