| Hello, On Aug 11, 2009, at 8:15 AM, Ralph Castain wrote: You can turn off those mca params I gave you as you are now past that point. I know there are others that can help debug that TCP btl error, but they can help you there. Just to eliminate the mitgcm from the debugging I compiled example/hello_c.c and run as: /usr/local/openmpi/bin/mpirun --debug-daemons -n 8 -host xserve01 hello_c >& hello_c4_1host.txt There is no ostensible problem. If I run as: /usr/local/openmpi/bin/mpirun --debug-daemons -n 8 -host xserve01,xserve02 hello_c >& hello_c4_2host.txt The process says Hello, but hangs at the end, and needs to be killed with ^C. I then modified connectivity_c to include a printf as MPI is initialized, and hardwired verbose=1. This completes, and appears to work fine.. /usr/local/openmpi/bin/mpirun --debug-daemons -n 8 -host xserve01 connectivity_c >& connectivity_c8_1host.txt However, again, two hosts sours the mix: /usr/local/openmpi/bin/mpirun --debug-daemons -n 8 -host xserve01,xserve02 connectivity_c >& connectivity_c8_2host.txt This hangs, and after waiting a minute or so we see that rank 0--4 on xserve01 cannot contact rank 5 (presumably on xserve02). It seems that I have something wrong in my tcp setup, but communication between these servers worked yesterday using 1.1.5, and ping etc all work fine, so something else is up. Some sort of port permissions? Th most glaring error I see in these is: [xserve02.local:43625] [[28627,0],2] orte:daemon:send_relay - recipient list is empty! I see reference in the archives to a similar error where "contacts.txt" could not be found. I've had trouble with 10.5.7 with temporary directories, so maybe that is the issue? Thanks Jody |
[saturna.cluster:19174] progressed_wait: base/plm_base_launch_support.c 459
Daemon was launched on xserve01.cluster - beginning to initialize
Daemon [[28401,0],1] checking in as pid 43824 on host xserve01.cluster
Daemon [[28401,0],1] not using static ports
[saturna.cluster:19174] defining message event: base/plm_base_launch_support.c
423
[xserve01.cluster:43824] [[28401,0],1] orted: up and running - waiting for
commands!
[saturna.cluster:19174] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19174] progressed_wait: base/plm_base_launch_support.c 712
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[saturna.cluster:19174] [[28401,0],0] node[0].name saturna daemon 0 arch
ffc90200
[saturna.cluster:19174] [[28401,0],0] node[1].name xserve01 daemon 1 arch
ffc90200
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received add_local_procs
[saturna.cluster:19174] defining message event: base/odls_base_default_fns.c
1219
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,0],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] node[0].name saturna daemon 0 arch
ffc90200
[xserve01.cluster:43824] [[28401,0],1] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received add_local_procs
[saturna.cluster:19174] defining message event: base/plm_base_launch_support.c
668
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],0]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],1]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],2]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],2] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],2]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],3]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],3] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],3]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],4]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],4] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],4]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],5]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],5] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],5]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],6]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],6] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],6]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174] defining message event: base/routed_base_receive.c 153
28401,1],7]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],7] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync+nidmap from
local proc [[28401,1],7]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824]
[[28401,0],1] orted_recv_cmd: received message from [[28401,1],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],2]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],2] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],3]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],3] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],4]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],4] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],6]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],6] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],5]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],5] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174] [[28401,0],0] orted_recv_cmd: received message from
[[28401,0],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],7] f[saturna.cluster:19174] defining message event:
orted/orted_comm.c 159
lective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174]
[[28401,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19174] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19174] [[28401,0],0] orted:comm:message_local_procs delivering
message to job [28401,1] tag 15
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,0],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43824] [[28401,0],1] orted:comm:message_local_procs
delivering message to job [28401,1] tag 15
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],2]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],2] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],3]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],3] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],4]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],4] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],7]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],7] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],6]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],6] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],5]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],5] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],0][saturna.cluster:19174] defining message event: orted/orted_comm.c
159
9
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824]
[[28401,0],1] orte:daemon:cmd:processor: proce[saturna.cluster:19174]
[[28401,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19174]
[[28401,0],0] orte:daemon:cmd:processor called by [[28401,0],1] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19174] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19174] [[28401,0],0] orted:comm:message_local_procs delivering
message to job [28401,1] tag 17
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,0],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43824] [[28401,0],1] orted:comm:message_local_procs
delivering message to job [28401,1] tag 17
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay - recipient list
is empty!
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 0 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 2 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 4 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 5 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 6 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 7 of 8
[saturna.cluster:19174] defining message event: iof_hnp_receive.c 227
Hello, world, I am 1 of 8
[28401,0],1] orted_recv_cmd: received message from [[28401,1],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1]
orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],4]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1]
orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],4] for[saturna.cluster:19174] [[28401,0],0] orted_recv_cmd: reissued
recv
llective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],2]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],2] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],3]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],3] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],5]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],5] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[saturna.cluster:19174]
[[28401,0],0] orte:daemon:cmd:processor called by [[28401,0],1] for tag 1
serve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],6]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],6] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],7]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],7] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174]
[[28401,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19174] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19174] [[28401,0],0] orted:comm:message_local_procs delivering
message to job [28401,1] tag 17
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,0],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43824] [[28401,0],1] orted:comm:message_local_procs
delivering message to job [28401,1] tag 17
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],0]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],0]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],3]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],3] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],3]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],4]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],4] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],4]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],5]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],5] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],5]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],6]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],6] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],6]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],7]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],7] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],7]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],1]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],1]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message from
[[28401,1],2]
[xserve01.cluster:43824] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,1],2] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv: received sync from local
proc [[28401,1],2]
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] defining message event: iof_orted_read.c 211
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],1] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received iof_complete cmd
[saturna.cluster:19174] defining message event: base/plm_base_receive.c 327
commands completed
[saturna.cluster:19174]
[[28401,0],0] calling job_complete trigger
[saturna.cluster:19174] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19174] [[28401,0],0] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[saturna.cluster:19174] [[28401,0],0] orted_cmd: received exit
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay
[saturna.cluster:19174] [[28401,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: received message
from[saturna.cluster:19174] [[28401,0],0] calling orted_exit trigger
rted/orted_comm.c 159
[xserve01.cluster:43824] [[28401,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:cmd:processor called by
[[28401,0],0] for tag 1
[xserve01.cluster:43824] [[28401,0],1] orted_cmd: received exit
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay
[xserve01.cluster:43824] [[28401,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43824] [[28401,0],1] calling orted_shutdown trigger
[xserve01.cluster:43824] [[28401,0],1] orted: finalizing
[saturna.cluster:19193] progressed_wait: base/plm_base_launch_support.c 459
Daemon was launched on xserve01.cluster - beginning to initialize
Daemon [[28398,0],1] checking in as pid 43859 on host xserve01.cluster
Daemon [[28398,0],1] not using static ports
[saturna.cluster:19193] defining message event: base/plm_base_launch_support.c
423
[xserve01.cluster:43859] [[28398,0],1] orted: up and running - waiting for
commands!
Daemon was launched on xserve03.local - beginning to initialize
Daemon [[28398,0],2] checking in as pid 41698 on host xserve03.local
Daemon [[28398,0],2] not using static ports
[saturna.cluster:19193] defining message event: base/plm_base_launch_support.c
423
[xserve03.local:41698] [[28398,0],2] orted: up and running - waiting for
commands!
[saturna.cluster:19193] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19193] progressed_wait: base/plm_base_launch_support.c 712
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[saturna.cluster:19193] [[28398,0],0] node[0].name saturna daemon 0 arch
ffc90200
[saturna.cluster:19193] [[28398,0],0] node[1].name xserve01 daemon 1 arch
ffc90200
[saturna.cluster:19193] [[28398,0],0] node[2].name xserve03 daemon 2 arch
ffc90200
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received add_local_procs
[saturna.cluster:19193] defining message event: base/odls_base_default_fns.c
1219
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,0],0]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve01.cluster:43859] [[28398,0],1] node[0].name saturna daemon 0 arch
ffc90200
[xserve01.cluster:43859] [[28398,0],1] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve01.cluster:43859] [[28398,0],1] node[2].name xserve03 daemon 2 arch
ffc90200
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received add_local_procs
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,0],0]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve03.local:41698] [[28398,0],2] node[0].name saturna daemon 0 arch ffc90200
[xserve03.local:41698] [[28398,0],2] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve03.local:41698] [[28398,0],2] node[2].name xserve03 daemon 2 arch
ffc90200
[xserve03.local:41698] [[28398,0],2] orted_cmd: received add_local_procs
[saturna.cluster:19193] defining message event: base/plm_base_launch_support.c
668
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],2]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],2] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_recv: received sync+nidmap from
local proc [[28398,1],2]
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],0]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],0] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_recv: received sync+nidmap from
local proc [[28398,1],0]
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] defining message event: base/plm_base_launch_support.c
668
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay - recipient list is
empty!
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],6]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],6] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_recv: received sync+nidmap from
local proc [[28398,1],6]
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],4]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],4] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_recv: received sync+nidmap from
local proc [[28398,1],4]
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859]
[[28398,0],1] orted_recv_cmd: received message from [[28398,1],2]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],2] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],0]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],0] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],6]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],6] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],4]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c
15[saturna.cluster:19193] [[28398,0],0] orted_recv_cmd: reissued recv
cv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],4] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193]
[[28398,0],0] orte:daemon:cmd:processor called by [[28398,0],1] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],1]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],1] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_recv: received sync+nidmap from
local proc [[28398,1],1]
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],3]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[saturna.cluster:19193] defining message event: base/routed_base_receive.c 153
8,1],3] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_recv: received sync+nidmap from
local proc [[28398,1],3]
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],5]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],5] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_recv: received sync+nidmap from
local proc [[28398,1],5]
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],7]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698]
[[28398,0],2] orte:daemon:cmd:processor called by [[28398,1],7] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_recv: received sync+nidmap from
local proc [[28398,1],7]
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],1]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],1] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],5]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],5] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],3]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],3] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] [[28398,0],0] orted_recv_cmd: received message from
[[28398,0],2]
xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],7] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands comple[saturna.cluster:19193] defining message event:
orted/orted_comm.c 159
[saturna.cluster:19193]
[[28398,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],2] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19193] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19193] [[28398,0],0] orted:comm:message_local_procs delivering
message to job [28398,1] tag 15
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,0],0]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received message_local_procs
[xserve03.local:41698] [[28398,0],2] orted:comm:message_local_procs delivering
message to job [28398,1] tag 15
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay - recipient list is
empty!
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,0],0]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43859] [[28398,0],1] orted:comm:message_local_procs
delivering message to job [28398,1] tag 15
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],3]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],3] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],6]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,1],6] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,1],4]
[xserve03.local:41698]
[[28398,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv[xserve01.cluster:43859]
[[28398,0],1] orte:daemon:cmd:processor called by [[28398,1],4] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor: processing
commands completed
1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698]
[[28398,0],2] orte:daemon:cmd:processor: processing commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],7]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],7] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,1],1]
[[saturna.cluster:19193] defining message event: orted/orted_comm.c 159
serve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,1],1] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received collective data cmd
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor: processing
commands completed
s completed
[saturna.cluster:19193]
[[28398,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19193] [[28398,0],0] orted_recv_cmd: received message from
[[28398,0],2]
[saturna.cluster:19193] defining message event: orted/orted_comm.c 159
[saturna.cluster:19193] [[28398,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],1] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],2] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19193] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[saturna.cluster:19193] [[28398,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19193] [[28398,0],0] orted:comm:message_local_procs delivering
message to job [28398,1] tag 17
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19193] [[28398,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: received message from
[[28398,0],0]
[xserve01.cluster:43859] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43859] [[28398,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve01.cluster:43859] [[28398,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43859] [[28398,0],1] orted:comm:message_local_procs
delivering message to job [28398,1] tag 17
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay
[xserve01.cluster:43859] [[28398,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: received message from
[[28398,0],0]
[xserve03.local:41698] defining message event: orted/orted_comm.c 159
[xserve03.local:41698] [[28398,0],2] orted_recv_cmd: reissued recv
[xserve03.local:41698] [[28398,0],2] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[xserve03.local:41698] [[28398,0],2] orted_cmd: received message_local_procs
[xserve03.local:41698] [[28398,0],2] orted:comm:message_local_procs delivering
message to job [28398,1] tag 17
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay
[xserve03.local:41698] [[28398,0],2] orte:daemon:send_relay - recipient list is
empty!
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 0 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 6 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 2 of 8
Hello, world, I am 4 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 3 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 1 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
Hello, world, I am 5 of 8
Hello, world, I am 7 of 8
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
[xserve03.local][[28398,1],1][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack]
received unexpected process identifier [[28398,1],2]
[saturna.cluster:19193] defining message event: iof_hnp_receive.c 227
[xserve01.cluster][[28398,1],0][btl_tcp_endpoint.c:486:mca_btl_tcp_endpoint_recv_connect_ack]
received unexpected process identifier [[28398,1],5]
[saturna.cluster:19193] defining timer event: 0 sec 0 usec at orterun.c:1128
Killed by signal 2.
.
[saturna.cluster:19193] [[28398,0],0]:orterun.c(1031)
updating exit status to 1
[saturna.cluster:19193] defining message event: base/plm_base_orted_cmds.c 276
[saturna.cluster:19193] defining timeout: 0 sec 2000 usec
at base/plm_base_orted_cmds.c:321
[saturna.cluster:19193] progressed_wait: base/plm_base_orted_cmds.c 324
[saturna.cluster:19193] defining timeout: 0 sec 8000 usec at orterun.c:1066
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor called by
[[28398,0],0] for tag 1
[saturna.cluster:19193] defining message event: base/odls_base_default_fns.c
2267
[saturna.cluster:19193] [[28398,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19193] [[28398,0],0] calling orted_exit trigger
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
--------------------------------------------------------------------------
xserve01
xserve03
[saturna.cluster:19236] progressed_wait: base/plm_base_launch_support.c 459
Daemon was launched on xserve01.cluster - beginning to initialize
Daemon [[28467,0],1] checking in as pid 43911 on host xserve01.cluster
Daemon [[28467,0],1] not using static ports
[saturna.cluster:19236] defining message event: base/plm_base_launch_support.c
423
!
[saturna.cluster:19236]
defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19236] progressed_wait: base/plm_base_launch_support.c 712
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[saturna.cluster:19236] [[28467,0],0] node[0].name saturna daemon 0 arch
ffc90200
[saturna.cluster:19236] [[28467,0],0] node[1].name xserve01 daemon 1 arch
ffc90200
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received add_local_procs
[saturna.cluster:19236] defining message event: base/odls_base_default_fns.c
1219
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,0],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] node[0].name saturna daemon 0 arch
ffc90200
[xserve01.cluster:43911] [[28467,0],1] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received add_local_procs
[saturna.cluster:19236] defining message event: base/plm_base_launch_support.c
668
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list
is empty!
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
MPI init
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],0]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],1]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],1]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],2]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],2] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],2]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],3]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],3] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],3]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],4]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],4] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],4]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],5]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],5] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],5]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],6]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],6] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],6]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236] defining message event: base/routed_base_receive.c 153
28467,1],7]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],7] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync+nidmap from
local proc [[28467,1],7]
[xserve01.cluster:43911]
[[28467,0],1] orte:daemon:cmd:processor: processing commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],2]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],2] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],1]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],3]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],3] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],5]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],5] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],4]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],4] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],6]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],6] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: received message from
[[28467,0],1]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],7] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911]
[[28467,0],1] orte:daemon:cmd:processor: proce[saturna.cluster:19236]
[[28467,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19236]
[[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering
message to job [28467,1] tag 15
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,0],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs
delivering message to job [28467,1] tag 15
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],5]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],5] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],2]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],2] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],3]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],3] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],4]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],4] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],6]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],6] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],7]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],7] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],0][saturna.cluster:19236] defining message event: orted/orted_comm.c
159
9
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],1]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911]
[[28467,0],1] orte:daemon:cmd:processor: proce[saturna.cluster:19236]
[[28467,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19236]
[[28467,0],0] orte:daemon:cmd:processor called by [[28467,0],1] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering
message to job [28467,1] tag 17
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,0],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs
delivering message to job [28467,1] tag 17
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list
is empty!
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
checking connection between rank 0 on xserve01.cluster and rank 1
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 2
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve01.cluster and rank 2
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 3
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve01.cluster and rank 3
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 2 on xserve01.cluster and rank 3
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 4
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve01.cluster and rank 4
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 2 on xserve01.cluster and rank 4
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 3 on xserve01.cluster and rank 4
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 5
checking connection between rank 0 on xserve01.cluster and rank 6
: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],1]
[xserve01.cluster:43911] defining message event[saturna.cluster:19236] defining
message event: iof_hnp_receive.c 227
recv_cmd: reissued recv
[xserve01.cluster:43911]
[[28467,0],1] orte:daemon:cmd:processor call[saturna.cluster:19236] defining
message event: iof_hnp_receive.c 227
rted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911]
[[28467,0],1] orted_recv_cmd: received messa[saturna.cluster:19236] defining
message event: iof_hnp_receive.c 227
orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],2] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],3]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],3] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],4]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],4] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],5]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],5] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],6]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
checking
connection between rank 2 on xserve01.cluster and rank 5
checking connection between rank 2 on xserve01.cluster and rank 6
ed by [[28467,1],6] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],7]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],7] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236]
defining message event: iof_hnp_receive.c 227
checking connection between rank 3 on xserve01.cluster and rank 5
checking connection between rank 3 on xserve01.cluster and rank 6
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 4 on xserve01.cluster and rank 5
checking connection between rank 4 on xserve01.cluster and rank 6
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 2 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 3 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 5 on xserve01.cluster and rank 6
checking connection between rank 5 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 4 on xserve01.cluster and rank 7
[saturna.cluster:19236] defining message event: iof_hnp_receive.c 227
checking connection between rank 6 on xserve01.cluster and rank 7
[saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: received message from
[[28467,0],1]
[saturna.cluster:19236] defining message event: orted/orted_comm.c 159
[saturna.cluster:19236] [[28467,0],0] orted_recv_cmd: reissued recv
Connectivity test on 8 processes PASSED.
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19236] [[28467,0],0] orted:comm:message_local_procs delivering
message to job [28467,1] tag 17
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,0],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:43911] [[28467,0],1] orted:comm:message_local_procs
delivering message to job [28467,1] tag 17
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],0]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],0]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],2]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],2] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],2]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],4]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],4] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],4]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],5]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],5] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],5]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],6]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],6] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],6]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],7]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],7] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],7]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],3]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],3] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],3]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message from
[[28467,1],1]
[xserve01.cluster:43911] defining message event: orted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,1],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv: received sync from local
proc [[28467,1],1]
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: base/odls_base_default_fns.c
2055
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] defining message event: iof_orted_read.c 211
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received waitpid_fired cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],1] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received iof_complete cmd
[saturna.cluster:19236] defining message event: base/plm_base_receive.c 327
commands completed
[saturna.cluster:19236]
[[28467,0],0] calling job_complete trigger
[saturna.cluster:19236] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19236] [[28467,0],0] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[saturna.cluster:19236] [[28467,0],0] orted_cmd: received exit
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay
[saturna.cluster:19236] [[28467,0],0] orte:daemon:send_relay sending relay msg
to 1
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: received message
from[saturna.cluster:19236] [[28467,0],0] calling orted_exit trigger
rted/orted_comm.c 159
[xserve01.cluster:43911] [[28467,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:cmd:processor called by
[[28467,0],0] for tag 1
[xserve01.cluster:43911] [[28467,0],1] orted_cmd: received exit
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay
[xserve01.cluster:43911] [[28467,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve01.cluster:43911] [[28467,0],1] calling orted_shutdown trigger
[xserve01.cluster:43911] [[28467,0],1] orted: finalizing
[saturna.cluster:19337] progressed_wait: base/plm_base_launch_support.c 459
Daemon was launched on xserve02.local - beginning to initialize
Daemon was launched on xserve01.cluster - beginning to initialize
Daemon [[28574,0],2] checking in as pid 43537 on host xserve02.local
Daemon [[28574,0],2] not using static ports
[saturna.cluster:19337] defining message event: base/plm_base_launch_support.c
423
[xserve02.local:43537] [[28574,0],2] orted: up and running - waiting for
commands!
Daemon [[28574,0],1] checking in as pid 44056 on host xserve01.cluster
Daemon [[28574,0],1] not using static ports
[saturna.cluster:19337] defining message event: base/plm_base_launch_support.c
423
[xserve01.cluster:44056] [[28574,0],1] orted: up and running - waiting for
commands!
[saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19337] progressed_wait: base/plm_base_launch_support.c 712
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[saturna.cluster:19337] [[28574,0],0] node[0].name saturna daemon 0 arch
ffc90200
[saturna.cluster:19337] [[28574,0],0] node[1].name xserve01 daemon 1 arch
ffc90200
[saturna.cluster:19337] [[28574,0],0] node[2].name xserve02 daemon 2 arch
ffc90200
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received add_local_procs
[saturna.cluster:19337] defining message event: base/odls_base_default_fns.c
1219
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,0],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] node[0].name saturna daemon 0 arch
ffc90200
[xserve01.cluster:44056] [[28574,0],1] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve01.cluster:44056] [[28574,0],1] node[2].name xserve02 daemon 2 arch
ffc90200
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received add_local_procs
[[28574,0],0]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve02.local:43537] [[28574,0],2] node[0].name saturna daemon 0 arch ffc90200
[xserve02.local:43537] [[28574,0],2] node[1].name xserve01 daemon 1 arch
ffc90200
[xserve02.local:43537] [[28574,0],2] node[2].name xserve02 daemon 2 arch
ffc90200
[xserve02.local:43537] [[28574,0],2] orted_cmd: received add_local_procs
[xserve02.local:43537]
[[28574,0],2] orte:daemon:send_relay
[xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is
empty!
serve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay - recipient list is
empty!
[saturna.cluster:19337]
defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: base/plm_base_launch_support.c
668
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
MPI init
MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
MPI init
MPI init
MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
MPI init
MPI init
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from
local proc [[28574,1],0]
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],1]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],1] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from
local proc [[28574,1],1]
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],2]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],2] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from
local proc [[28574,1],2]
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],6]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],6] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from
local proc [[28574,1],6]
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],4]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],4] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_recv: received sync+nidmap from
local proc [[28574,1],4]
[saturna.cluster:19337] defining message event: base/routed_base_receive.c 153
mmands completed
[xserve01.cluster:44056]
[[28574,0],1] orted_recv_cmd: received message from [[28574,1],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],6]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],6] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],2]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],2] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: received message from
[[28574,0],1]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],4] f[saturna.cluster:19337] defining message event:
orted/orted_comm.c 159
lective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337]
[[28574,0],0] orted_recv_cmd: reissued recv
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by
[[28574,0],1] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],3]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],3] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from
local proc [[28574,1],3]
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],7]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],7] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from
local proc [[28574,1],7]
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],5]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],5] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_recv: received sync+nidmap from
local proc [[28574,1],5]
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537]
[[28574,0],2] orted_recv_cmd: received message from [[28574,1],1]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],1] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],5]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],5] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],3]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],3] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],7]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[x[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv
erve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],7] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337]
[[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],2] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19337] [[28574,0],0] orted:comm:message_local_procs delivering
message to job [28574,1] tag 15
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,0],0]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received message_local_procs
[xserve02.local:43537] [[28574,0],2] orted:comm:message_local_procs delivering
message to job [28574,1] tag 15
[xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay
[xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is
empty!
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,0],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:44056] [[28574,0],1] orted:comm:message_local_procs
delivering message to job [28574,1] tag 15
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:send_relay - recipient list
is empty!
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],5]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],5] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],3]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],3] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],7]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],7] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],6]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],6] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],2]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],2] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received collective data cmd
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor: processing
commands completed
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,1],4]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,1],4] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received
col[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv
cmd:processor: processing commands completed
[saturna.cluster:19337]
[[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],1] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing
commands completed
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,1],1]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[x[saturna.cluster:19337] [[28574,0],0] orted_recv_cmd: reissued recv
erve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,1],1] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received collective data cmd
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337]
[[28574,0],0] orte:daemon:cmd:processor called by [[28574,0],2] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received collective data cmd
[saturna.cluster:19337] defining message event: grpcomm_bad_module.c 183
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[saturna.cluster:19337] [[28574,0],0] orted_cmd: received message_local_procs
[saturna.cluster:19337] [[28574,0],0] orted:comm:message_local_procs delivering
message to job [28574,1] tag 17
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 1
[saturna.cluster:19337] [[28574,0],0] orte:daemon:send_relay sending relay msg
to 2
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: received message from
[[28574,0],0]
[xserve02.local:43537] defining message event: orted/orted_comm.c 159
[xserve02.local:43537] [[28574,0],2] orted_recv_cmd: reissued recv
[xserve02.local:43537] [[28574,0],2] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve02.local:43537] [[28574,0],2] orted_cmd: received message_local_procs
[xserve02.local:43537] [[28574,0],2] orted:comm:message_local_procs delivering
message to job [28574,1] tag 17
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: received message from
[[28574,0],0]
[xserve01.cluster:44056] defining message event: orted/orted_comm.c 159
[xserve01.cluster:44056] [[28574,0],1] orted_recv_cmd: reissued recv
[xserve01.cluster:44056] [[28574,0],1] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[xserve01.cluster:44056] [[28574,0],1] orted_cmd: received message_local_procs
[xserve01.cluster:44056] [[28574,0],1] orted:comm:message_local_procs
delivering message to job [28574,1] tag 17
[xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay
[xserve02.local:43537] [[28574,0],2] orte:daemon:send_relay - recipient list is
empty!
ty!
[saturna.cluster:19337]
defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
Done MPI init
checking connection between rank 0 on xserve01.cluster and rank 1
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
Done MPI init
Done MPI init
Done MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
Done MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
Done MPI init
Done MPI init
Done MPI init
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve02.local and rank 2
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 2
checking connection between rank 0 on xserve01.cluster and rank 3
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 2 on xserve01.cluster and rank 3
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve02.local and rank 3
checking connection between rank 1 on xserve02.local and rank 4
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 0 on xserve01.cluster and rank 4
checking connection between rank 0 on xserve01.cluster and rank 5
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 3 on xserve02.local and rank 4
checking connection between rank 2 on xserve01.cluster and rank 4
checking connection between rank 2 on xserve01.cluster and rank 5
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 1 on xserve02.local and rank 5
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 4 on xserve01.cluster and rank 5
[saturna.cluster:19337] defining message event: iof_hnp_receive.c 227
checking connection between rank 3 on xserve02.local and rank 5
Killed by signal 2.
7] defining timer event: 0 sec 0 usec at orterun.c:1128
mpirun: killing job...
[saturna.cluster:19337]
[[28574,0],0]:orterun.c(1031) updating exit status to 1
[saturna.cluster:19337] defining message event: base/plm_base_orted_cmds.c 276
[saturna.cluster:19337] defining timeout: 0 sec 2000 usec at
base/plm_base_orted_cmds.c:321
[saturna.cluster:19337] progressed_wait: base/plm_base_orted_cmds.c 324
[saturna.cluster:19337] defining timeout: 0 sec 8000 usec at orterun.c:1066
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor called by
[[28574,0],0] for tag 1
[saturna.cluster:19337] defining message event: base/odls_base_default_fns.c
2267
[saturna.cluster:19337] [[28574,0],0] orte:daemon:cmd:processor: processing
commands completed
[saturna.cluster:19337] [[28574,0],0] calling orted_exit trigger
--------------------------------------------------------------------------
mpirun was unable to cleanly terminate the daemons on the nodes shown
below. Additional manual cleanup may be required - please refer to
the "orte-clean" tool for assistance.
--------------------------------------------------------------------------
xserve01
xserve02
