No need for the lstopo data anymore, Bill - I was able to recreate the situation using some very nice hwloc functions plus your prior descriptions. I'm not totally confident that this fix will resolve the problem but it will clear out at least one problem.
We'll just have to see what happens and attack it next. Ralph On Tue, Jul 7, 2015 at 8:07 PM, Lane, William <william.l...@cshs.org> wrote: > I'm sorry I haven't been able to get the lstopo information for > all the nodes, but I had to get the latest version of hwloc installed > first. They've even added in some more modern blades that also > support hyperthreading, ugh. They've also been doing some memory > upgrades as well. > > I'm trying to get a Bash script running on the cluster via qsub > that will run lstopo and output the host information to a file located > in my $HOME directory but it hasn't been working (there are 60 nodes > in the heterogeneous cluster that needs to have OpenMPI running). > > I will try to get the lstopo information by the end of the week. > > I'd be willing to do most anything to get these OpenMPI issues > resolved. I'd even wash your cars for you! > > -Bill L. > ________________________________________ > From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain [ > r...@open-mpi.org] > Sent: Tuesday, July 07, 2015 1:36 PM > To: Open MPI Users > Subject: Re: [OMPI users] OpenMPI 1.8.6, CentOS 6.3, too many slots = crash > > I may have finally tracked this down. At least, I can now get the correct > devel map to come out, and found a memory corruption issue that only > impacted hetero operations. I can’t know if this is the root cause of the > problem Bill is seeing, however, as I have no way of actually running the > job. > > I pushed this into the master and will bring it back to 1.8.7 as well as > 1.10. > > Bill - would you be able/willing to give it a try there? It would be nice > to confirm this actually fixed the problem. > > > > On Jun 29, 2015, at 1:58 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> > wrote: > > > > lstopo will tell you -- if there is more than one "PU" (hwloc > terminology for "processing unit") per core, then hyper threading is > enabled. If there's only one PU per core, then hyper threading is disabled. > > > > > >> On Jun 29, 2015, at 4:42 PM, Lane, William <william.l...@cshs.org> > wrote: > >> > >> Would the output of dmidecode -t processor and/or lstopo tell me > conclusively > >> if hyperthreading is enabled or not? Hyperthreading is supposed to be > enabled > >> for all the IBM x3550 M3 and M4 nodes, but I'm not sure if it actually > is and I > >> don't have access to the BIOS settings. > >> > >> -Bill L. > >> > >> From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain [ > r...@open-mpi.org] > >> Sent: Saturday, June 27, 2015 7:21 PM > >> To: Open MPI Users > >> Subject: Re: [OMPI users] OpenMPI 1.8.6, CentOS 6.3, too many slots = > crash > >> > >> Bill - this is such a jumbled collection of machines that I’m having > trouble figuring out what I should replicate. I can create something > artificial here so I can try to debug this, but I need to know exactly what > I’m up against - can you tell me: > >> > >> * the architecture of each type - how many sockets, how many > cores/socket, HT on or off. If two nodes have the same physical setup but > one has HT on and the other off, then please consider those two different > types > >> > >> * how many nodes of each type > >> > >> Looking at your map output, it looks like the map is being done > correctly, but somehow the binding locale isn’t getting set in some cases. > You latest error output would seem out-of-step with your prior reports, so > something else may be going on there. As I said earlier, this is the most > hetero environment we’ve seen, and so there may be some code paths your > hitting that haven’t been well exercised before. > >> > >> > >> > >> > >>> On Jun 26, 2015, at 5:22 PM, Lane, William <william.l...@cshs.org> > wrote: > >>> > >>> Well, I managed to get a successful mpirun @ a slot count of 132 using > --mca btl ^sm, > >>> however when I increased the slot count to 160, mpirun crashed without > any error > >>> output: > >>> > >>> mpirun -np 160 -display-devel-map --prefix > /hpc/apps/mpi/openmpi/1.8.6/ --hostfile hostfile-noslots --mca btl ^sm > --hetero-nodes --bind-to core /hpc/home/lanew/mpi/openmpi/ProcessColors3 >> > out.txt 2>&1 > >>> > >>> > -------------------------------------------------------------------------- > >>> WARNING: a request was made to bind a process. While the system > >>> supports binding the process itself, at least one node does NOT > >>> support binding memory to the process location. > >>> > >>> Node: csclprd3-6-1 > >>> > >>> This usually is due to not having the required NUMA support installed > >>> on the node. In some Linux distributions, the required support is > >>> contained in the libnumactl and libnumactl-devel packages. > >>> This is a warning only; your job will continue, though performance may > be degraded. > >>> > -------------------------------------------------------------------------- > >>> > -------------------------------------------------------------------------- > >>> A request was made to bind to that would result in binding more > >>> processes than cpus on a resource: > >>> > >>> Bind to: CORE > >>> Node: csclprd3-6-1 > >>> #processes: 2 > >>> #cpus: 1 > >>> > >>> You can override this protection by adding the "overload-allowed" > >>> option to your binding directive. > >>> > -------------------------------------------------------------------------- > >>> > >>> But csclprd3-6-1 (a blade) does have 2 CPU's on 2 separate sockets w/2 > cores apiece as shown in my dmidecode output: > >>> > >>> csclprd3-6-1 ~]# dmidecode -t processor > >>> # dmidecode 2.11 > >>> SMBIOS 2.4 present. > >>> > >>> Handle 0x0008, DMI type 4, 32 bytes > >>> Processor Information > >>> Socket Designation: Socket 1 CPU 1 > >>> Type: Central Processor > >>> Family: Xeon > >>> Manufacturer: GenuineIntel > >>> ID: F6 06 00 00 01 03 00 00 > >>> Signature: Type 0, Family 6, Model 15, Stepping 6 > >>> Flags: > >>> FPU (Floating-point unit on-chip) > >>> CX8 (CMPXCHG8 instruction supported) > >>> APIC (On-chip APIC hardware supported) > >>> Version: Intel Xeon > >>> Voltage: 2.9 V > >>> External Clock: 333 MHz > >>> Max Speed: 4000 MHz > >>> Current Speed: 3000 MHz > >>> Status: Populated, Enabled > >>> Upgrade: ZIF Socket > >>> L1 Cache Handle: 0x0004 > >>> L2 Cache Handle: 0x0005 > >>> L3 Cache Handle: Not Provided > >>> > >>> Handle 0x0009, DMI type 4, 32 bytes > >>> Processor Information > >>> Socket Designation: Socket 2 CPU 2 > >>> Type: Central Processor > >>> Family: Xeon > >>> Manufacturer: GenuineIntel > >>> ID: F6 06 00 00 01 03 00 00 > >>> Signature: Type 0, Family 6, Model 15, Stepping 6 > >>> Flags: > >>> FPU (Floating-point unit on-chip) > >>> CX8 (CMPXCHG8 instruction supported) > >>> APIC (On-chip APIC hardware supported) > >>> Version: Intel Xeon > >>> Voltage: 2.9 V > >>> External Clock: 333 MHz > >>> Max Speed: 4000 MHz > >>> Current Speed: 3000 MHz > >>> Status: Populated, Enabled > >>> Upgrade: ZIF Socket > >>> L1 Cache Handle: 0x0006 > >>> L2 Cache Handle: 0x0007 > >>> L3 Cache Handle: Not Provided > >>> > >>> csclprd3-6-1 ~]# lstopo > >>> Machine (16GB) > >>> Socket L#0 + L2 L#0 (4096KB) > >>> L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0) > >>> L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2) > >>> Socket L#1 + L2 L#1 (4096KB) > >>> L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#1) > >>> L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#3) > >>> > >>> csclprd3-0-1 information (which looks correct as this particular x3550 > should > >>> have one socket populated (of two) with a 6 core Xeon (or 12 cores > w/hyperthreading > >>> turned on): > >>> > >>> csclprd3-0-1 ~]# lstopo > >>> Machine (71GB) > >>> Socket L#0 + L3 L#0 (12MB) > >>> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + > PU L#0 (P#0) > >>> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + > PU L#1 (P#1) > >>> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + > PU L#2 (P#2) > >>> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + > PU L#3 (P#3) > >>> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + > PU L#4 (P#4) > >>> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + > PU L#5 (P#5) > >>> > >>> csclprd3-0-1 ~]# dmidecode -t processor > >>> # dmidecode 2.11 > >>> # SMBIOS entry point at 0x7f6be000 > >>> SMBIOS 2.5 present. > >>> > >>> Handle 0x0001, DMI type 4, 40 bytes > >>> Processor Information > >>> Socket Designation: Node 1 Socket 1 > >>> Type: Central Processor > >>> Family: Xeon MP > >>> Manufacturer: Intel(R) Corporation > >>> ID: C2 06 02 00 FF FB EB BF > >>> Signature: Type 0, Family 6, Model 44, Stepping 2 > >>> Flags: > >>> FPU (Floating-point unit on-chip) > >>> VME (Virtual mode extension) > >>> DE (Debugging extension) > >>> PSE (Page size extension) > >>> TSC (Time stamp counter) > >>> MSR (Model specific registers) > >>> PAE (Physical address extension) > >>> MCE (Machine check exception) > >>> CX8 (CMPXCHG8 instruction supported) > >>> APIC (On-chip APIC hardware supported) > >>> SEP (Fast system call) > >>> MTRR (Memory type range registers) > >>> PGE (Page global enable) > >>> MCA (Machine check architecture) > >>> CMOV (Conditional move instruction supported) > >>> PAT (Page attribute table) > >>> PSE-36 (36-bit page size extension) > >>> CLFSH (CLFLUSH instruction supported) > >>> DS (Debug store) > >>> ACPI (ACPI supported) > >>> MMX (MMX technology supported) > >>> FXSR (FXSAVE and FXSTOR instructions supported) > >>> SSE (Streaming SIMD extensions) > >>> SSE2 (Streaming SIMD extensions 2) > >>> SS (Self-snoop) > >>> HTT (Multi-threading) > >>> TM (Thermal monitor supported) > >>> PBE (Pending break enabled) > >>> Version: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > >>> Voltage: 1.2 V > >>> External Clock: 5866 MHz > >>> Max Speed: 4400 MHz > >>> Current Speed: 2400 MHz > >>> Status: Populated, Enabled > >>> Upgrade: ZIF Socket > >>> L1 Cache Handle: 0x0002 > >>> L2 Cache Handle: 0x0003 > >>> L3 Cache Handle: 0x0004 > >>> Serial Number: Not Specified > >>> Asset Tag: Not Specified > >>> Part Number: Not Specified > >>> Core Count: 6 > >>> Core Enabled: 6 > >>> Thread Count: 6 > >>> Characteristics: > >>> 64-bit capable > >>> > >>> Handle 0x005A, DMI type 4, 40 bytes > >>> Processor Information > >>> Socket Designation: Node 1 Socket 2 > >>> Type: Central Processor > >>> Family: Xeon MP > >>> Manufacturer: Not Specified > >>> ID: 00 00 00 00 00 00 00 00 > >>> Signature: Type 0, Family 0, Model 0, Stepping 0 > >>> Flags: None > >>> Version: Not Specified > >>> Voltage: 1.2 V > >>> External Clock: 5866 MHz > >>> Max Speed: 4400 MHz > >>> Current Speed: Unknown > >>> Status: Unpopulated > >>> Upgrade: ZIF Socket > >>> L1 Cache Handle: Not Provided > >>> L2 Cache Handle: Not Provided > >>> L3 Cache Handle: Not Provided > >>> Serial Number: Not Specified > >>> Asset Tag: Not Specified > >>> Part Number: Not Specified > >>> Characteristics: None > >>> > >>> > >>> From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain [ > r...@open-mpi.org] > >>> Sent: Wednesday, June 24, 2015 6:06 AM > >>> To: Open MPI Users > >>> Subject: Re: [OMPI users] OpenMPI 1.8.6, CentOS 6.3, too many slots = > crash > >>> > >>> I think trying with --mca btl ^sm makes a lot of sense and may solve > the problem. I also noted that we are having trouble with the topology of > several of the nodes - seeing only one socket, non-HT where you say we > should see two sockets and HT-enabled. In those cases, the locality is > "unknown" - given that those procs are on remote nodes from the one being > impacted, I don't think it should cause a problem. However, it isn't > correct, and that raises flags. > >>> > >>> My best guess of the root cause of that error is either we are getting > bad topology info on those nodes, or we have a bug that is mishandling this > scenario. It would probably be good to get this error fixed to ensure it > isn't the source of the eventual crash, even though I'm not sure they are > related. > >>> > >>> Bill: Can we examine one of the problem nodes? Let's pick csclprd3-0-1 > (or take another one from your list - just look for one where "locality" is > reported as "unknown" for the procs in the output map). Can you run lstopo > on that node and send us the output? In the above map, it is reporting a > single socket with 6 cores, non-HT. Is that what lstopo shows when run on > the node? Is it what you expected? > >>> > >>> > >>> On Wed, Jun 24, 2015 at 4:07 AM, Gilles Gouaillardet < > gilles.gouaillar...@gmail.com> wrote: > >>> Bill, > >>> > >>> were you able to get a core file and analyze the stack with gdb ? > >>> > >>> I suspect the error occurs in mca_btl_sm_add_procs but this is just my > best guess. > >>> if this is correct, can you check the value of > mca_btl_sm_component.num_smp_procs ? > >>> > >>> as a workaround, can you try > >>> mpirun --mca btl ^sm ... > >>> > >>> I do not see how I can tackle the root cause without being able to > reproduce the issue :-( > >>> > >>> can you try to reproduce the issue with the smallest hostfile, and > then run lstopo on all the nodes ? > >>> btw, you are not mixing 32 bits and 64 bits OS, are you ? > >>> > >>> Cheers, > >>> > >>> Gilles > >>> > >>> > >>> > >>> > >>> mca_btl_sm_add_procs( > >>> > >>> > >>> > >>> int > >>> > >>> mca_btl_sm_add_procs > >>> ( > >>> On Wednesday, June 24, 2015, Lane, William <william.l...@cshs.org> > wrote: > >>> Gilles, > >>> > >>> All the blades only have two core Xeons (without hyperthreading) > populating both their sockets. All > >>> the x3550 nodes have hyperthreading capable Xeons and Sandybridge > server CPU's. It's possible > >>> hyperthreading has been disabled on some of these nodes though. The > 3-0-n nodes are all IBM x3550 > >>> nodes while the 3-6-n nodes are all blade nodes. > >>> > >>> I have run this exact same test code successfully in the past on > another cluster (~200 nodes of Sunfire X2100 > >>> 2x dual-core Opterons) w/no issues on upwards of 390 slots. I even > tested it recently on OpenMPI 1.8.5 > >>> on another smaller R&D cluster consisting of 10 Sunfire X2100 nodes > (w/2 dual core Opterons apiece). > >>> On this particular cluster I've had success running this code on < 132 > slots. > >>> > >>> Anyway, here's the results of the following mpirun: > >>> > >>> mpirun -np 132 -display-devel-map --prefix > /hpc/apps/mpi/openmpi/1.8.6/ --hostfile hostfile-noslots --mca > btl_tcp_if_include eth0 --hetero-nodes --bind-to core > /hpc/home/lanew/mpi/openmpi/ProcessColors3 >> out.txt 2>&1 > >>> > >>> > -------------------------------------------------------------------------- > >>> WARNING: a request was made to bind a process. While the system > >>> supports binding the process itself, at least one node does NOT > >>> support binding memory to the process location. > >>> > >>> Node: csclprd3-6-1 > >>> > >>> This usually is due to not having the required NUMA support installed > >>> on the node. In some Linux distributions, the required support is > >>> contained in the libnumactl and libnumactl-devel packages. > >>> This is a warning only; your job will continue, though performance may > be degraded. > >>> > -------------------------------------------------------------------------- > >>> Data for JOB [51718,1] offset 0 > >>> > >>> Mapper requested: NULL Last mapper: round_robin Mapping policy: > BYSOCKET Ranking policy: SLOT > >>> Binding policy: CORE Cpu set: NULL PPR: NULL Cpus-per-rank: 1 > >>> Num new daemons: 0 New daemon starting vpid INVALID > >>> Num nodes: 15 > >>> > >>> Data for node: csclprd3-6-1 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],1] Daemon launched: True > >>> Num slots: 4 Slots in use: 4 Oversubscribed: FALSE > >>> Num slots allocated: 4 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 4 Next node_rank: 4 > >>> Data for proc: [[51718,1],0] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 0 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B][./.] > >>> Binding: [B/.][./.] > >>> Data for proc: [[51718,1],1] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 1 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./.][B/B] > >>> Binding: [./.][B/.] > >>> Data for proc: [[51718,1],2] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 2 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B][./.] > >>> Binding: [./B][./.] > >>> Data for proc: [[51718,1],3] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 3 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./.][B/B] > >>> Binding: [./.][./B] > >>> > >>> Data for node: csclprd3-6-5 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],2] Daemon launched: True > >>> Num slots: 4 Slots in use: 4 Oversubscribed: FALSE > >>> Num slots allocated: 4 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 4 Next node_rank: 4 > >>> Data for proc: [[51718,1],4] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 4 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B][./.] > >>> Binding: [B/.][./.] > >>> Data for proc: [[51718,1],5] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 5 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./.][B/B] > >>> Binding: [./.][B/.] > >>> Data for proc: [[51718,1],6] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 6 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B][./.] > >>> Binding: [./B][./.] > >>> Data for proc: [[51718,1],7] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 7 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./.][B/B] > >>> Binding: [./.][./B] > >>> > >>> Data for node: csclprd3-0-0 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],3] Daemon launched: True > >>> Num slots: 12 Slots in use: 12 Oversubscribed: FALSE > >>> Num slots allocated: 12 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 12 Next node_rank: 12 > >>> Data for proc: [[51718,1],8] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 8 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [B/././././.][./././././.] > >>> Data for proc: [[51718,1],9] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 9 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][B/././././.] > >>> Data for proc: [[51718,1],10] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 10 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [./B/./././.][./././././.] > >>> Data for proc: [[51718,1],11] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 11 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][./B/./././.] > >>> Data for proc: [[51718,1],12] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 12 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [././B/././.][./././././.] > >>> Data for proc: [[51718,1],13] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 13 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][././B/././.] > >>> Data for proc: [[51718,1],14] > >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 14 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [./././B/./.][./././././.] > >>> Data for proc: [[51718,1],15] > >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 15 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][./././B/./.] > >>> Data for proc: [[51718,1],16] > >>> Pid: 0 Local rank: 8 Node rank: 8 App rank: 16 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [././././B/.][./././././.] > >>> Data for proc: [[51718,1],17] > >>> Pid: 0 Local rank: 9 Node rank: 9 App rank: 17 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][././././B/.] > >>> Data for proc: [[51718,1],18] > >>> Pid: 0 Local rank: 10 Node rank: 10 App rank: 18 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [B/B/B/B/B/B][./././././.] > >>> Binding: [./././././B][./././././.] > >>> Data for proc: [[51718,1],19] > >>> Pid: 0 Local rank: 11 Node rank: 11 App rank: 19 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [./././././.][B/B/B/B/B/B] > >>> Binding: [./././././.][./././././B] > >>> > >>> Data for node: csclprd3-0-1 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],4] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],20] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 20 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],21] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 21 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],22] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 22 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],23] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 23 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],24] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 24 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],25] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 25 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-2 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],5] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],26] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 26 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],27] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 27 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],28] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 28 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],29] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 29 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],30] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 30 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],31] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 31 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-3 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],6] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],32] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 32 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],33] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 33 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],34] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 34 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],35] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 35 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],36] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 36 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],37] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 37 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-4 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],7] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],38] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 38 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],39] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 39 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],40] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 40 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],41] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 41 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],42] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 42 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],43] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 43 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-5 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],8] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],44] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 44 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],45] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 45 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],46] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 46 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],47] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 47 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],48] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 48 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],49] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 49 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-6 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],9] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],50] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 50 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [B/././././.] > >>> Data for proc: [[51718,1],51] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 51 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./B/./././.] > >>> Data for proc: [[51718,1],52] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 52 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././B/././.] > >>> Data for proc: [[51718,1],53] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 53 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././B/./.] > >>> Data for proc: [[51718,1],54] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 54 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [././././B/.] > >>> Data for proc: [[51718,1],55] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 55 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [./././././B] > >>> > >>> Data for node: csclprd3-0-7 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],10] Daemon launched: True > >>> Num slots: 16 Slots in use: 16 Oversubscribed: FALSE > >>> Num slots allocated: 16 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 16 Next node_rank: 16 > >>> Data for proc: [[51718,1],56] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 56 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [BB/../../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],57] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 57 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][BB/../../../../../../..] > >>> Data for proc: [[51718,1],58] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 58 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../BB/../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],59] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 59 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../BB/../../../../../..] > >>> Data for proc: [[51718,1],60] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 60 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../BB/../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],61] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 61 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../BB/../../../../..] > >>> Data for proc: [[51718,1],62] > >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 62 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../BB/../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],63] > >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 63 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../BB/../../../..] > >>> Data for proc: [[51718,1],64] > >>> Pid: 0 Local rank: 8 Node rank: 8 App rank: 64 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../BB/../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],65] > >>> Pid: 0 Local rank: 9 Node rank: 9 App rank: 65 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../BB/../../..] > >>> Data for proc: [[51718,1],66] > >>> Pid: 0 Local rank: 10 Node rank: 10 App rank: 66 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../BB/../..][../../../../../../../..] > >>> Data for proc: [[51718,1],67] > >>> Pid: 0 Local rank: 11 Node rank: 11 App rank: 67 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../BB/../..] > >>> Data for proc: [[51718,1],68] > >>> Pid: 0 Local rank: 12 Node rank: 12 App rank: 68 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../BB/..][../../../../../../../..] > >>> Data for proc: [[51718,1],69] > >>> Pid: 0 Local rank: 13 Node rank: 13 App rank: 69 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../BB/..] > >>> Data for proc: [[51718,1],70] > >>> Pid: 0 Local rank: 14 Node rank: 14 App rank: 70 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../../BB][../../../../../../../..] > >>> Data for proc: [[51718,1],71] > >>> Pid: 0 Local rank: 15 Node rank: 15 App rank: 71 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../../BB] > >>> > >>> Data for node: csclprd3-0-8 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],11] Daemon launched: True > >>> Num slots: 16 Slots in use: 16 Oversubscribed: FALSE > >>> Num slots allocated: 16 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 16 Next node_rank: 16 > >>> Data for proc: [[51718,1],72] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 72 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [BB/../../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],73] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 73 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][BB/../../../../../../..] > >>> Data for proc: [[51718,1],74] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 74 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../BB/../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],75] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 75 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../BB/../../../../../..] > >>> Data for proc: [[51718,1],76] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 76 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../BB/../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],77] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 77 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../BB/../../../../..] > >>> Data for proc: [[51718,1],78] > >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 78 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../BB/../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],79] > >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 79 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../BB/../../../..] > >>> Data for proc: [[51718,1],80] > >>> Pid: 0 Local rank: 8 Node rank: 8 App rank: 80 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../BB/../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],81] > >>> Pid: 0 Local rank: 9 Node rank: 9 App rank: 81 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../BB/../../..] > >>> Data for proc: [[51718,1],82] > >>> Pid: 0 Local rank: 10 Node rank: 10 App rank: 82 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../BB/../..][../../../../../../../..] > >>> Data for proc: [[51718,1],83] > >>> Pid: 0 Local rank: 11 Node rank: 11 App rank: 83 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../BB/../..] > >>> Data for proc: [[51718,1],84] > >>> Pid: 0 Local rank: 12 Node rank: 12 App rank: 84 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../BB/..][../../../../../../../..] > >>> Data for proc: [[51718,1],85] > >>> Pid: 0 Local rank: 13 Node rank: 13 App rank: 85 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../BB/..] > >>> Data for proc: [[51718,1],86] > >>> Pid: 0 Local rank: 14 Node rank: 14 App rank: 86 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../../BB][../../../../../../../..] > >>> Data for proc: [[51718,1],87] > >>> Pid: 0 Local rank: 15 Node rank: 15 App rank: 87 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../../BB] > >>> > >>> Data for node: csclprd3-0-10 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],12] Daemon launched: True > >>> Num slots: 16 Slots in use: 16 Oversubscribed: FALSE > >>> Num slots allocated: 16 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 16 Next node_rank: 16 > >>> Data for proc: [[51718,1],88] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 88 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [BB/../../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],89] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 89 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][BB/../../../../../../..] > >>> Data for proc: [[51718,1],90] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 90 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../BB/../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],91] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 91 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../BB/../../../../../..] > >>> Data for proc: [[51718,1],92] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 92 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../BB/../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],93] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 93 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../BB/../../../../..] > >>> Data for proc: [[51718,1],94] > >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 94 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../BB/../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],95] > >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 95 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../BB/../../../..] > >>> Data for proc: [[51718,1],96] > >>> Pid: 0 Local rank: 8 Node rank: 8 App rank: 96 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../BB/../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],97] > >>> Pid: 0 Local rank: 9 Node rank: 9 App rank: 97 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../BB/../../..] > >>> Data for proc: [[51718,1],98] > >>> Pid: 0 Local rank: 10 Node rank: 10 App rank: 98 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../BB/../..][../../../../../../../..] > >>> Data for proc: [[51718,1],99] > >>> Pid: 0 Local rank: 11 Node rank: 11 App rank: 99 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../BB/../..] > >>> Data for proc: [[51718,1],100] > >>> Pid: 0 Local rank: 12 Node rank: 12 App rank: 100 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../BB/..][../../../../../../../..] > >>> Data for proc: [[51718,1],101] > >>> Pid: 0 Local rank: 13 Node rank: 13 App rank: 101 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../BB/..] > >>> Data for proc: [[51718,1],102] > >>> Pid: 0 Local rank: 14 Node rank: 14 App rank: 102 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../../BB][../../../../../../../..] > >>> Data for proc: [[51718,1],103] > >>> Pid: 0 Local rank: 15 Node rank: 15 App rank: 103 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../../BB] > >>> > >>> Data for node: csclprd3-0-11 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],13] Daemon launched: True > >>> Num slots: 16 Slots in use: 16 Oversubscribed: FALSE > >>> Num slots allocated: 16 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 16 Next node_rank: 16 > >>> Data for proc: [[51718,1],104] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 104 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [BB/../../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],105] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 105 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][BB/../../../../../../..] > >>> Data for proc: [[51718,1],106] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 106 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../BB/../../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],107] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 107 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../BB/../../../../../..] > >>> Data for proc: [[51718,1],108] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 108 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../BB/../../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],109] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 109 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../BB/../../../../..] > >>> Data for proc: [[51718,1],110] > >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 110 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../BB/../../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],111] > >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 111 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../BB/../../../..] > >>> Data for proc: [[51718,1],112] > >>> Pid: 0 Local rank: 8 Node rank: 8 App rank: 112 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../BB/../../..][../../../../../../../..] > >>> Data for proc: [[51718,1],113] > >>> Pid: 0 Local rank: 9 Node rank: 9 App rank: 113 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../BB/../../..] > >>> Data for proc: [[51718,1],114] > >>> Pid: 0 Local rank: 10 Node rank: 10 App rank: 114 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../BB/../..][../../../../../../../..] > >>> Data for proc: [[51718,1],115] > >>> Pid: 0 Local rank: 11 Node rank: 11 App rank: 115 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../BB/../..] > >>> Data for proc: [[51718,1],116] > >>> Pid: 0 Local rank: 12 Node rank: 12 App rank: 116 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../BB/..][../../../../../../../..] > >>> Data for proc: [[51718,1],117] > >>> Pid: 0 Local rank: 13 Node rank: 13 App rank: 117 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../BB/..] > >>> Data for proc: [[51718,1],118] > >>> Pid: 0 Local rank: 14 Node rank: 14 App rank: 118 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> Binding: [../../../../../../../BB][../../../../../../../..] > >>> Data for proc: [[51718,1],119] > >>> Pid: 0 Local rank: 15 Node rank: 15 App rank: 119 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../../../..][../../../../../../../BB] > >>> > >>> Data for node: csclprd3-0-12 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],14] Daemon launched: True > >>> Num slots: 6 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 6 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],120] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 120 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [BB/../../../../..] > >>> Data for proc: [[51718,1],121] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 121 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [../BB/../../../..] > >>> Data for proc: [[51718,1],122] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 122 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [../../BB/../../..] > >>> Data for proc: [[51718,1],123] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 123 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [../../../BB/../..] > >>> Data for proc: [[51718,1],124] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 124 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [../../../../BB/..] > >>> Data for proc: [[51718,1],125] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 125 > >>> State: INITIALIZED App_context: 0 > >>> Locale: UNKNOWN > >>> Binding: [../../../../../BB] > >>> > >>> Data for node: csclprd3-0-13 Launch id: -1 State: 0 > >>> Daemon: [[51718,0],15] Daemon launched: True > >>> Num slots: 12 Slots in use: 6 Oversubscribed: FALSE > >>> Num slots allocated: 12 Max slots: 0 > >>> Username on node: NULL > >>> Num procs: 6 Next node_rank: 6 > >>> Data for proc: [[51718,1],126] > >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 126 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB][../../../../../..] > >>> Binding: [BB/../../../../..][../../../../../..] > >>> Data for proc: [[51718,1],127] > >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 127 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../..][BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../..][BB/../../../../..] > >>> Data for proc: [[51718,1],128] > >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 128 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB][../../../../../..] > >>> Binding: [../BB/../../../..][../../../../../..] > >>> Data for proc: [[51718,1],129] > >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 129 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../..][BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../..][../BB/../../../..] > >>> Data for proc: [[51718,1],130] > >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 130 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [BB/BB/BB/BB/BB/BB][../../../../../..] > >>> Binding: [../../BB/../../..][../../../../../..] > >>> Data for proc: [[51718,1],131] > >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 131 > >>> State: INITIALIZED App_context: 0 > >>> Locale: [../../../../../..][BB/BB/BB/BB/BB/BB] > >>> Binding: [../../../../../..][../../BB/../../..] > >>> [csclprd3-0-13:31619] *** Process received signal *** > >>> [csclprd3-0-13:31619] Signal: Bus error (7) > >>> [csclprd3-0-13:31619] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31619] Failing at address: 0x7f1374267a00 > >>> [csclprd3-0-13:31620] *** Process received signal *** > >>> [csclprd3-0-13:31620] Signal: Bus error (7) > >>> [csclprd3-0-13:31620] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31620] Failing at address: 0x7fcc702a7980 > >>> [csclprd3-0-13:31615] *** Process received signal *** > >>> [csclprd3-0-13:31615] Signal: Bus error (7) > >>> [csclprd3-0-13:31615] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31615] Failing at address: 0x7f8128367880 > >>> [csclprd3-0-13:31616] *** Process received signal *** > >>> [csclprd3-0-13:31616] Signal: Bus error (7) > >>> [csclprd3-0-13:31616] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31616] Failing at address: 0x7fe674227a00 > >>> [csclprd3-0-13:31617] *** Process received signal *** > >>> [csclprd3-0-13:31617] Signal: Bus error (7) > >>> [csclprd3-0-13:31617] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31617] Failing at address: 0x7f061c32db80 > >>> [csclprd3-0-13:31618] *** Process received signal *** > >>> [csclprd3-0-13:31618] Signal: Bus error (7) > >>> [csclprd3-0-13:31618] Signal code: Non-existant physical address (2) > >>> [csclprd3-0-13:31618] Failing at address: 0x7fb8402eaa80 > >>> [csclprd3-0-13:31618] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7fb851851500] > >>> [csclprd3-0-13:31618] [ 1] [csclprd3-0-13:31616] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7fe6843a4500] > >>> [csclprd3-0-13:31616] [ 1] [csclprd3-0-13:31620] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7fcc80c54500] > >>> [csclprd3-0-13:31620] [ 1] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7fcc80fc9f61] > >>> [csclprd3-0-13:31620] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7fcc80fca047] > >>> [csclprd3-0-13:31620] [ 3] [csclprd3-0-13:31615] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7f81385ca500] > >>> [csclprd3-0-13:31615] [ 1] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7f813893ff61] > >>> [csclprd3-0-13:31615] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7f8138940047] > >>> [csclprd3-0-13:31615] [ 3] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7fb851bc6f61] > >>> [csclprd3-0-13:31618] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7fb851bc7047] > >>> [csclprd3-0-13:31618] [ 3] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7fb851ab4670] > >>> [csclprd3-0-13:31618] [ 4] [csclprd3-0-13:31617] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7f062cfe5500] > >>> [csclprd3-0-13:31617] [ 1] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7f062d35af61] > >>> [csclprd3-0-13:31617] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7f062d35b047] > >>> [csclprd3-0-13:31617] [ 3] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7f062d248670] > >>> [csclprd3-0-13:31617] [ 4] [csclprd3-0-13:31619] [ 0] > /lib64/libpthread.so.0(+0xf500)[0x7f1384fde500] > >>> [csclprd3-0-13:31619] [ 1] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7f1385353f61] > >>> [csclprd3-0-13:31619] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x167f61)[0x7fe684719f61] > >>> [csclprd3-0-13:31616] [ 2] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7fe68471a047] > >>> [csclprd3-0-13:31616] [ 3] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7fe684607670] > >>> [csclprd3-0-13:31616] [ 4] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x168047)[0x7f1385354047] > >>> [csclprd3-0-13:31619] [ 3] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7f1385241670] > >>> [csclprd3-0-13:31619] [ 4] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7f13852425ab] > >>> [csclprd3-0-13:31619] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7f1385242751] > >>> [csclprd3-0-13:31619] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7f13853501c9] > >>> [csclprd3-0-13:31619] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7f1385336628] > >>> [csclprd3-0-13:31619] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7fcc80eb7670] > >>> [csclprd3-0-13:31620] [ 4] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7fcc80eb85ab] > >>> [csclprd3-0-13:31620] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7fcc80eb8751] > >>> [csclprd3-0-13:31620] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7fcc80fc61c9] > >>> [csclprd3-0-13:31620] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7fcc80fac628] > >>> [csclprd3-0-13:31620] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7fcc8111fd61] > >>> [csclprd3-0-13:31620] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x55670)[0x7f813882d670] > >>> [csclprd3-0-13:31615] [ 4] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7f813882e5ab] > >>> [csclprd3-0-13:31615] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7f813882e751] > >>> [csclprd3-0-13:31615] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7f813893c1c9] > >>> [csclprd3-0-13:31615] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7f8138922628] > >>> [csclprd3-0-13:31615] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7f8138a95d61] > >>> [csclprd3-0-13:31615] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7f813885d747] > >>> [csclprd3-0-13:31615] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7fb851ab55ab] > >>> [csclprd3-0-13:31618] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7fb851ab5751] > >>> [csclprd3-0-13:31618] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7fb851bc31c9] > >>> [csclprd3-0-13:31618] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7fb851ba9628] > >>> [csclprd3-0-13:31618] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7fb851d1cd61] > >>> [csclprd3-0-13:31618] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7fb851ae4747] > >>> [csclprd3-0-13:31618] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7f062d2495ab] > >>> [csclprd3-0-13:31617] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7f062d249751] > >>> [csclprd3-0-13:31617] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7f062d3571c9] > >>> [csclprd3-0-13:31617] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7f062d33d628] > >>> [csclprd3-0-13:31617] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7f062d4b0d61] > >>> [csclprd3-0-13:31617] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7f062d278747] > >>> [csclprd3-0-13:31617] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_grow+0x3b9)[0x7fe6846085ab] > >>> [csclprd3-0-13:31616] [ 5] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_free_list_resize_mt+0xfb)[0x7fe684608751] > >>> [csclprd3-0-13:31616] [ 6] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_btl_sm_add_procs+0x671)[0x7fe6847161c9] > >>> [csclprd3-0-13:31616] [ 7] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(+0x14a628)[0x7fe6846fc628] > >>> [csclprd3-0-13:31616] [ 8] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7fe68486fd61] > >>> [csclprd3-0-13:31616] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7fe684637747] > >>> [csclprd3-0-13:31616] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7fe68467750b] > >>> [csclprd3-0-13:31616] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31616] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fe684021cdd] > >>> [csclprd3-0-13:31616] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31616] *** End of error message *** > >>> > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7f062d2b850b] > >>> [csclprd3-0-13:31617] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31617] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f062cc62cdd] > >>> [csclprd3-0-13:31617] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31617] *** End of error message *** > >>> > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(mca_pml_ob1_add_procs+0xff)[0x7f13854a9d61] > >>> [csclprd3-0-13:31619] [ 9] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7f1385271747] > >>> [csclprd3-0-13:31619] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7f13852b150b] > >>> [csclprd3-0-13:31619] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31619] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1384c5bcdd] > >>> [csclprd3-0-13:31619] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31619] *** End of error message *** > >>> > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(ompi_mpi_init+0xbda)[0x7fcc80ee7747] > >>> [csclprd3-0-13:31620] [10] > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7fcc80f2750b] > >>> [csclprd3-0-13:31620] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31620] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fcc808d1cdd] > >>> [csclprd3-0-13:31620] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31620] *** End of error message *** > >>> > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7f813889d50b] > >>> [csclprd3-0-13:31615] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31615] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8138247cdd] > >>> [csclprd3-0-13:31615] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31615] *** End of error message *** > >>> > /hpc/apps/mpi/openmpi/1.8.6/lib/libmpi.so.1(MPI_Init+0x185)[0x7fb851b2450b] > >>> [csclprd3-0-13:31618] [11] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] > >>> [csclprd3-0-13:31618] [12] > /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fb8514cecdd] > >>> [csclprd3-0-13:31618] [13] > /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] > >>> [csclprd3-0-13:31618] *** End of error message *** > >>> > -------------------------------------------------------------------------- > >>> mpirun noticed that process rank 126 with PID 0 on node csclprd3-0-13 > exited on signal 7 (Bus error). > >>> > -------------------------------------------------------------------------- > >>> > >>> From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain [ > r...@open-mpi.org] > >>> Sent: Tuesday, June 23, 2015 6:20 PM > >>> To: Open MPI Users > >>> Subject: Re: [OMPI users] OpenMPI 1.8.6, CentOS 6.3, too many slots = > crash > >>> > >>> Wow - that is one sick puppy! I see that some nodes are reporting > not-bound for their procs, and the rest are binding to socket (as they > should). Some of your nodes clearly do not have hyper threads enabled (or > only have single-thread cores on them), and have 2 cores/socket. Other > nodes have 8 cores/socket with hyper threads enabled, while still others > have 6 cores/socket and HT enabled. > >>> > >>> I don't see anyone binding to a single HT if multiple HTs/core are > available. I think you are being fooled by those nodes that either don't > have HT enabled, or have only 1 HT/core. > >>> > >>> In both cases, it is node 13 that is the node that fails. I also note > that you said everything works okay with < 132 ranks, and node 13 hosts > ranks 127-131. So node 13 would host ranks even if you reduced the number > in the job to 131. This would imply that it probably isn't something wrong > with the node itself. > >>> > >>> Is there any way you could run a job of this size on a homogeneous > cluster? The procs all show bindings that look right, but I'm wondering if > the heterogeneity is the source of the trouble here. We may be > communicating the binding pattern incorrectly and giving bad info to the > backend daemon. > >>> > >>> Also, rather than --report-bindings, use "--display-devel-map" on the > command line and let's see what the mapper thinks it did. If there is a > problem with placement, that is where it would exist. > >>> > >>> > >>> On Tue, Jun 23, 2015 at 5:12 PM, Lane, William <william.l...@cshs.org> > wrote: > >>> Ralph, > >>> > >>> There is something funny going on, the trace from the > >>> runs w/the debug build aren't showing any differences from > >>> what I got earlier. However, I did do a run w/the --bind-to core > >>> switch and was surprised to see that hyperthreading cores were > >>> sometimes being used. > >>> > >>> Here's the traces that I have: > >>> > >>> mpirun -np 132 -report-bindings --prefix /hpc/apps/mpi/openmpi/1.8.6/ > --hostfile hostfile-noslots --mca btl_tcp_if_include eth0 --hetero-nodes > /hpc/home/lanew/mpi/openmpi/ProcessColors3 > >>> [csclprd3-0-5:16802] MCW rank 44 is not bound (or bound to all > available processors) > >>> [csclprd3-0-5:16802] MCW rank 45 is not bound (or bound to all > available processors) > >>> [csclprd3-0-5:16802] MCW rank 46 is not bound (or bound to all > available processors) > >>> [csclprd3-6-5:12480] MCW rank 4 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]]: [B/B][./.] > >>> [csclprd3-6-5:12480] MCW rank 5 bound to socket 1[core 2[hwt 0]], > socket 1[core 3[hwt 0]]: [./.][B/B] > >>> [csclprd3-6-5:12480] MCW rank 6 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]]: [B/B][./.] > >>> [csclprd3-6-5:12480] MCW rank 7 bound to socket 1[core 2[hwt 0]], > socket 1[core 3[hwt 0]]: [./.][B/B] > >>> [csclprd3-0-5:16802] MCW rank 47 is not bound (or bound to all > available processors) > >>> [csclprd3-0-5:16802] MCW rank 48 is not bound (or bound to all > available processors) > >>> [csclprd3-0-5:16802] MCW rank 49 is not bound (or bound to all > available processors) > >>> [csclprd3-0-1:14318] MCW rank 22 is not bound (or bound to all > available processors) > >>> [csclprd3-0-1:14318] MCW rank 23 is not bound (or bound to all > available processors) > >>> [csclprd3-0-1:14318] MCW rank 24 is not bound (or bound to all > available processors) > >>> [csclprd3-6-1:24682] MCW rank 3 bound to socket 1[core 2[hwt 0]], > socket 1[core 3[hwt 0]]: [./.][B/B] > >>> [csclprd3-6-1:24682] MCW rank 0 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]]: [B/B][./.] > >>> [csclprd3-0-1:14318] MCW rank 25 is not bound (or bound to all > available processors) > >>> [csclprd3-0-1:14318] MCW rank 20 is not bound (or bound to all > available processors) > >>> [csclprd3-0-3:13827] MCW rank 34 is not bound (or bound to all > available processors) > >>> [csclprd3-0-1:14318] MCW rank 21 is not bound (or bound to all > available processors) > >>> [csclprd3-0-3:13827] MCW rank 35 is not bound (or bound to all > available processors) > >>> [csclprd3-6-1:24682] MCW rank 1 bound to socket 1[core 2[hwt 0]], > socket 1[core 3[hwt 0]]: [./.][B/B] > >>> [csclprd3-0-3:13827] MCW rank 36 is not bound (or bound to all > available processors) > >>> [csclprd3-6-1:24682] MCW rank 2 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]]: [B/B][./.] > >>> [csclprd3-0-6:30371] MCW rank 51 is not bound (or bound to all > available processors) > >>> [csclprd3-0-6:30371] MCW rank 52 is not bound (or bound to all > available processors) > >>> [csclprd3-0-6:30371] MCW rank 53 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 30 is not bound (or bound to all > available processors) > >>> [csclprd3-0-6:30371] MCW rank 54 is not bound (or bound to all > available processors) > >>> [csclprd3-0-3:13827] MCW rank 37 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 31 is not bound (or bound to all > available processors) > >>> [csclprd3-0-3:13827] MCW rank 32 is not bound (or bound to all > available processors) > >>> [csclprd3-0-6:30371] MCW rank 55 is not bound (or bound to all > available processors) > >>> [csclprd3-0-3:13827] MCW rank 33 is not bound (or bound to all > available processors) > >>> [csclprd3-0-6:30371] MCW rank 50 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 26 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 27 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 28 is not bound (or bound to all > available processors) > >>> [csclprd3-0-2:05825] MCW rank 29 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 121 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 122 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 123 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 124 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 125 is not bound (or bound to all > available processors) > >>> [csclprd3-0-12:12383] MCW rank 120 is not bound (or bound to all > available processors) > >>> [csclprd3-0-0:31079] MCW rank 13 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-0:31079] MCW rank 14 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-0:31079] MCW rank 15 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-0:31079] MCW rank 16 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-7:20515] MCW rank 68 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-10:19096] MCW rank 100 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-7:20515] MCW rank 69 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-10:19096] MCW rank 101 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-0:31079] MCW rank 17 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-7:20515] MCW rank 70 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-10:19096] MCW rank 102 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-11:31636] MCW rank 116 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-11:31636] MCW rank 117 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-0:31079] MCW rank 18 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-11:31636] MCW rank 118 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-0:31079] MCW rank 19 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-7:20515] MCW rank 71 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-10:19096] MCW rank 103 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-0:31079] MCW rank 8 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-0:31079] MCW rank 9 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-10:19096] MCW rank 88 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-11:31636] MCW rank 119 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-7:20515] MCW rank 56 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-0:31079] MCW rank 10 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-7:20515] MCW rank 57 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-10:19096] MCW rank 89 bound to socket 1[core 8[hwt 0-1]], > socket 1[core 9[hwt 0-1]], socket 1[core 10[hwt 0-1]], socket 1[core 11[hwt > 0-1]], socket 1[core 12[hwt 0-1]], socket 1[core 13[hwt 0-1]], socket > 1[core 14[hwt 0-1]], socket 1[core 15[hwt 0-1]]: > [../../../../../../../..][BB/BB/BB/BB/BB/BB/BB/BB] > >>> [csclprd3-0-11:31636] MCW rank 104 bound to socket 0[core 0[hwt 0-1]], > socket 0[core 1[hwt 0-1]], socket 0[core 2[hwt 0-1]], socket 0[core 3[hwt > 0-1]], socket 0[core 4[hwt 0-1]], socket 0[core 5[hwt 0-1]], socket 0[core > 6[hwt 0-1]], socket 0[core 7[hwt 0-1]]: > [BB/BB/BB/BB/BB/BB/BB/BB][../../../../../../../..] > >>> [csclprd3-0-0:31079] MCW rank 11 bound to socket 1[core 6[hwt 0]], > socket 1[core 7[hwt 0]], socket 1[core 8[hwt 0]], socket 1[core 9[hwt 0]], > socket 1[core 10[hwt 0]], socket 1[core 11[hwt 0]]: > [./././././.][B/B/B/B/B/B] > >>> [csclprd3-0-0:31079] MCW rank 12 bound to socket 0[core 0[hwt 0]], > socket 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]], > socket 0[core 4[hwt 0]], socket 0[core 5[hwt 0]]: [B/B/B/B/B/B][./././././.] > >>> [csclprd3-0-4:30348] MCW rank 42 is not bound (or bound to all > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27185.php > >>> > >>> IMPORTANT WARNING: This message is intended for the use of the person > or entity to which it is addressed and may contain information that is > privileged and confidential, the disclosure of which is governed by > applicable law. If the reader of this message is not the intended > recipient, or the employee or agent responsible for delivering it to the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this information is strictly prohibited. Thank > you for your cooperation. > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27204.php > >> > >> IMPORTANT WARNING: This message is intended for the use of the person > or entity to which it is addressed and may contain information that is > privileged and confidential, the disclosure of which is governed by > applicable law. If the reader of this message is not the intended > recipient, or the employee or agent responsible for delivering it to the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this information is strictly prohibited. Thank > you for your cooperation. > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27220.php > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27222.php > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/07/27261.php > IMPORTANT WARNING: This message is intended for the use of the person or > entity to which it is addressed and may contain information that is > privileged and confidential, the disclosure of which is governed by > applicable law. If the reader of this message is not the intended > recipient, or the employee or agent responsible for delivering it to the > intended recipient, you are hereby notified that any dissemination, > distribution or copying of this information is strictly prohibited. Thank > you for your cooperation. > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/07/27263.php >
