I enclosure 2 files with output of two foloowing commands (OMPI 1.9a1r32570) $time mpirun --leave-session-attached -mca oob_base_verbose 100 -np 1 ./hello_c >& out1.txt (Hello, world, I am ....) real 1m3.952s user 0m0.035s sys 0m0.107s $time mpirun --leave-session-attached -mca oob_base_verbose 100 --mca oob_tcp_if_include ib0 -np 1 ./hello_c >& out2.txt (no Hello, word, I am ....) real 0m9.337s user 0m0.059s sys 0m0.098s Wed, 27 Aug 2014 06:31:02 -0700 от Ralph Castain <r...@open-mpi.org>: >How bizarre. Please add "--leave-session-attached -mca oob_base_verbose 100" >to your cmd line > >On Aug 27, 2014, at 4:31 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>When i try to specify oob with --mca oob_tcp_if_include <one of interface >>from ifconfig>, i alwase get error: >>$ mpirun --mca oob_tcp_if_include ib0 -np 1 ./hello_c >>-------------------------------------------------------------------------- >>An ORTE daemon has unexpectedly failed after launch and before >>communicating back to mpirun. This could be caused by a number >>of factors, including an inability to create a connection back >>to mpirun due to a lack of common network interfaces and/or no >>route found between them. Please check network connectivity >>(including firewalls and network routing requirements). >>------------------------------------------------------------------------- >> >>Earlier, in ompi 1.8.1, I can not run mpi jobs without " --mca >>oob_tcp_if_include ib0 "... but now(ompi 1.9.a1) with this flag i get above >>error. >> >>Here is an output of ifconfig >>$ ifconfig >>eth1 Link encap:Ethernet HWaddr 00:15:17:EE:89:E1 >>inet addr:10.0.251.53 Bcast:10.0.251.255 Mask:255.255.255.0 >>UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>RX packets:215087433 errors:0 dropped:0 overruns:0 frame:0 >>TX packets:2648 errors:0 dropped:0 overruns:0 carrier:0 >>collisions:0 txqueuelen:1000 >>RX bytes:26925754883 (25.0 GiB) TX bytes:137971 (134.7 KiB) >>Memory:b2c00000-b2c20000 >>eth2 Link encap:Ethernet HWaddr 00:02:C9:04:73:F8 >>inet addr:10.0.0.4 Bcast:10.0.0.255 Mask:255.255.255.0 >>UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>RX packets:4892833125 errors:0 dropped:0 overruns:0 frame:0 >>TX packets:8708606918 errors:0 dropped:0 overruns:0 carrier:0 >>collisions:0 txqueuelen:1000 >>RX bytes:1823986502132 (1.6 TiB) TX bytes:11957754120037 (10.8 TiB) >>eth2.911 Link encap:Ethernet HWaddr 00:02:C9:04:73:F8 >>inet addr:93.180.7.38 Bcast:93.180.7.63 Mask:255.255.255.224 >>UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>RX packets:3746454225 errors:0 dropped:0 overruns:0 frame:0 >>TX packets:1131917608 errors:0 dropped:3 overruns:0 carrier:0 >>collisions:0 txqueuelen:0 >>RX bytes:285174723322 (265.5 GiB) TX bytes:11523163526058 (10.4 TiB) >>eth3 Link encap:Ethernet HWaddr 00:02:C9:04:73:F9 >>inet addr:10.2.251.14 Bcast:10.2.251.255 Mask:255.255.255.0 >>UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>RX packets:591156692 errors:0 dropped:56 overruns:56 frame:56 >>TX packets:679729229 errors:0 dropped:0 overruns:0 carrier:0 >>collisions:0 txqueuelen:1000 >>RX bytes:324195989293 (301.9 GiB) TX bytes:770299202886 (717.3 GiB) >>Ifconfig uses the ioctl access method to get the full address information, >>which limits hardware addresses to 8 bytes. >>Because Infiniband address has 20 bytes, only the first 8 bytes are displayed >>correctly. >>Ifconfig is obsolete! For replacement check ip. >>ib0 Link encap:InfiniBand HWaddr >>80:00:00:48:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 >>inet addr:10.128.0.4 Bcast:10.128.255.255 Mask:255.255.0.0 >>UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 >>RX packets:10843859 errors:0 dropped:0 overruns:0 frame:0 >>TX packets:8089839 errors:0 dropped:15 overruns:0 carrier:0 >>collisions:0 txqueuelen:1024 >>RX bytes:939249464 (895.7 MiB) TX bytes:886054008 (845.0 MiB) >>lo Link encap:Local Loopback >>inet addr:127.0.0.1 Mask:255.0.0.0 >>UP LOOPBACK RUNNING MTU:16436 Metric:1 >>RX packets:31235107 errors:0 dropped:0 overruns:0 frame:0 >>TX packets:31235107 errors:0 dropped:0 overruns:0 carrier:0 >>collisions:0 txqueuelen:0 >>RX bytes:132750916041 (123.6 GiB) TX bytes:132750916041 (123.6 GiB) >> >> >> >>Tue, 26 Aug 2014 09:48:35 -0700 от Ralph Castain < r...@open-mpi.org >: >>>I think something may be messed up with your installation. I went ahead and >>>tested this on a Slurm 2.5.4 cluster, and got the following: >>> >>>$ time mpirun -np 1 --host bend001 ./hello >>>Hello, World, I am 0 of 1 [0 local peers]: get_cpubind: 0 bitmap 0,12 >>> >>>real 0m0.086s >>>user 0m0.039s >>>sys 0m0.046s >>> >>>$ time mpirun -np 1 --host bend002 ./hello >>>Hello, World, I am 0 of 1 [0 local peers]: get_cpubind: 0 bitmap 0,12 >>> >>>real 0m0.528s >>>user 0m0.021s >>>sys 0m0.023s >>> >>>Which is what I would have expected. With --host set to the local host, no >>>daemons are being launched and so the time is quite short (just spent >>>mapping and fork/exec). With --host set to a single remote host, you have >>>the time it takes Slurm to launch our daemon on the remote host, so you get >>>about half of a second. >>> >>>IIRC, you were having some problems with the OOB setup. If you specify the >>>TCP interface to use, does your time come down? >>> >>> >>>On Aug 26, 2014, at 8:32 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>>I'm using slurm 2.5.6 >>>> >>>>$salloc -N8 --exclusive -J ompi -p test >>>>$ srun hostname >>>>node1-128-21 >>>>node1-128-24 >>>>node1-128-22 >>>>node1-128-26 >>>>node1-128-27 >>>>node1-128-20 >>>>node1-128-25 >>>>node1-128-23 >>>>$ time mpirun -np 1 --host node1-128-21 ./hello_c >>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>semenov@compiler-2 Distribution, ident: 1.9a1r32570, repo rev: r32570, Aug >>>>21, 2014 (nightly snapshot tarball), 146) >>>>real 1m3.932s >>>>user 0m0.035s >>>>sys 0m0.072s >>>> >>>> >>>>Tue, 26 Aug 2014 07:03:58 -0700 от Ralph Castain < r...@open-mpi.org >: >>>>>hmmm....what is your allocation like? do you have a large hostfile, for >>>>>example? >>>>> >>>>>if you add a --host argument that contains just the local host, what is >>>>>the time for that scenario? >>>>> >>>>>On Aug 26, 2014, at 6:27 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>>>>Hello! >>>>>>Here is my time results: >>>>>>$time mpirun -n 1 ./hello_c >>>>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>>>semenov@compiler-2 Distribution, ident: 1.9a1r32570, repo rev: r32570, >>>>>>Aug 21, 2014 (nightly snapshot tarball), 146) >>>>>>real 1m3.985s >>>>>>user 0m0.031s >>>>>>sys 0m0.083s >>>>>> >>>>>> >>>>>>Fri, 22 Aug 2014 07:43:03 -0700 от Ralph Castain < r...@open-mpi.org >: >>>>>>>I'm also puzzled by your timing statement - I can't replicate it: >>>>>>> >>>>>>>07:41:43 $ time mpirun -n 1 ./hello_c >>>>>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>>>>rhc@bend001 Distribution, ident: 1.9a1r32577, repo rev: r32577, >>>>>>>Unreleased developer copy, 125) >>>>>>> >>>>>>>real 0m0.547s >>>>>>>user 0m0.043s >>>>>>>sys 0m0.046s >>>>>>> >>>>>>>The entire thing ran in 0.5 seconds >>>>>>> >>>>>>> >>>>>>>On Aug 22, 2014, at 6:33 AM, Mike Dubman < mi...@dev.mellanox.co.il > >>>>>>>wrote: >>>>>>>>Hi, >>>>>>>>The default delimiter is ";" . You can change delimiter with >>>>>>>>mca_base_env_list_delimiter. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>On Fri, Aug 22, 2014 at 2:59 PM, Timur Ismagilov < tismagi...@mail.ru >>>>>>>>> wrote: >>>>>>>>>Hello! >>>>>>>>>If i use latest night snapshot: >>>>>>>>>$ ompi_info -V >>>>>>>>>Open MPI v1.9a1r32570 >>>>>>>>>* In programm hello_c initialization takes ~1 min >>>>>>>>>In ompi 1.8.2rc4 and ealier it takes ~1 sec(or less) >>>>>>>>>* if i use >>>>>>>>>$mpirun --mca mca_base_env_list >>>>>>>>>'MXM_SHM_KCOPY_MODE=off,OMP_NUM_THREADS=8' --map-by slot:pe=8 -np 1 >>>>>>>>>./hello_c >>>>>>>>>i got error >>>>>>>>>config_parser.c:657 MXM ERROR Invalid value for SHM_KCOPY_MODE: >>>>>>>>>'off,OMP_NUM_THREADS=8'. Expected: [off|knem|cma|autodetect] >>>>>>>>>but with -x all works fine (but with warn) >>>>>>>>>$mpirun -x MXM_SHM_KCOPY_MODE=off -x OMP_NUM_THREADS=8 -np 1 ./hello_c >>>>>>>>>WARNING: The mechanism by which environment variables are explicitly >>>>>>>>>.............. >>>>>>>>>.............. >>>>>>>>>.............. >>>>>>>>>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>>>>>>>>semenov@compiler-2 Distribution, ident: 1.9a1r32570, repo rev: r32570, >>>>>>>>>Aug 21, 2014 (nightly snapshot tarball), 146) >>>>>>>>>Thu, 21 Aug 2014 06:26:13 -0700 от Ralph Castain < r...@open-mpi.org >: >>>>>>>>>>Not sure I understand. The problem has been fixed in both the trunk >>>>>>>>>>and the 1.8 branch now, so you should be able to work with either of >>>>>>>>>>those nightly builds. >>>>>>>>>> >>>>>>>>>>On Aug 21, 2014, at 12:02 AM, Timur Ismagilov < tismagi...@mail.ru > >>>>>>>>>>wrote: >>>>>>>>>>>Have i I any opportunity to run mpi jobs? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>Wed, 20 Aug 2014 10:48:38 -0700 от Ralph Castain < r...@open-mpi.org >>>>>>>>>>>>: >>>>>>>>>>>>yes, i know - it is cmr'd >>>>>>>>>>>> >>>>>>>>>>>>On Aug 20, 2014, at 10:26 AM, Mike Dubman < >>>>>>>>>>>>mi...@dev.mellanox.co.il > wrote: >>>>>>>>>>>>>btw, we get same error in v1.8 branch as well. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>On Wed, Aug 20, 2014 at 8:06 PM, Ralph Castain < >>>>>>>>>>>>>r...@open-mpi.org > wrote: >>>>>>>>>>>>>>It was not yet fixed - but should be now. >>>>>>>>>>>>>> >>>>>>>>>>>>>>On Aug 20, 2014, at 6:39 AM, Timur Ismagilov < tismagi...@mail.ru >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>Hello! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>As i can see, the bug is fixed, but in Open MPI v1.9a1r32516 i >>>>>>>>>>>>>>>still have the problem >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>a) >>>>>>>>>>>>>>>$ mpirun -np 1 ./hello_c >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>An ORTE daemon has unexpectedly failed after launch and before >>>>>>>>>>>>>>>communicating back to mpirun. This could be caused by a number >>>>>>>>>>>>>>>of factors, including an inability to create a connection back >>>>>>>>>>>>>>>to mpirun due to a lack of common network interfaces and/or no >>>>>>>>>>>>>>>route found between them. Please check network connectivity >>>>>>>>>>>>>>>(including firewalls and network routing requirements). >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>b) >>>>>>>>>>>>>>>$ mpirun --mca oob_tcp_if_include ib0 -np 1 ./hello_c >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>An ORTE daemon has unexpectedly failed after launch and before >>>>>>>>>>>>>>>communicating back to mpirun. This could be caused by a number >>>>>>>>>>>>>>>of factors, including an inability to create a connection back >>>>>>>>>>>>>>>to mpirun due to a lack of common network interfaces and/or no >>>>>>>>>>>>>>>route found between them. Please check network connectivity >>>>>>>>>>>>>>>(including firewalls and network routing requirements). >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>c) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>$ mpirun --mca oob_tcp_if_include ib0 -debug-daemons --mca >>>>>>>>>>>>>>>plm_base_verbose 5 -mca oob_base_verbose 10 -mca >>>>>>>>>>>>>>>rml_base_verbose 10 -np 1 ./hello_c >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Querying component >>>>>>>>>>>>>>>[isolated] >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Query of component >>>>>>>>>>>>>>>[isolated] set priority to 0 >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Querying component >>>>>>>>>>>>>>>[rsh] >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Query of component >>>>>>>>>>>>>>>[rsh] set priority to 10 >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Querying component >>>>>>>>>>>>>>>[slurm] >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Query of component >>>>>>>>>>>>>>>[slurm] set priority to 75 >>>>>>>>>>>>>>>[compiler-2:14673] mca:base:select:( plm) Selected component >>>>>>>>>>>>>>>[slurm] >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: registering >>>>>>>>>>>>>>>oob components >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: found loaded >>>>>>>>>>>>>>>component tcp >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: component tcp >>>>>>>>>>>>>>>register function successful >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: opening oob >>>>>>>>>>>>>>>components >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: found loaded >>>>>>>>>>>>>>>component tcp >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: component tcp >>>>>>>>>>>>>>>open function successful >>>>>>>>>>>>>>>[compiler-2:14673] mca:oob:select: checking available component >>>>>>>>>>>>>>>tcp >>>>>>>>>>>>>>>[compiler-2:14673] mca:oob:select: Querying component [tcp] >>>>>>>>>>>>>>>[compiler-2:14673] oob:tcp: component_available called >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] oob:tcp:init adding 10.128.0.4 >>>>>>>>>>>>>>>to our list of V4 connections >>>>>>>>>>>>>>>[compiler-2:14673] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] TCP STARTUP >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] attempting to bind to IPv4 port >>>>>>>>>>>>>>>0 >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] assigned IPv4 port 59460 >>>>>>>>>>>>>>>[compiler-2:14673] mca:oob:select: Adding component to end >>>>>>>>>>>>>>>[compiler-2:14673] mca:oob:select: Found 1 active transports >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: registering >>>>>>>>>>>>>>>rml components >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: found loaded >>>>>>>>>>>>>>>component oob >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_register: component oob >>>>>>>>>>>>>>>has no register or open function >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: opening rml >>>>>>>>>>>>>>>components >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: found loaded >>>>>>>>>>>>>>>component oob >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: components_open: component oob >>>>>>>>>>>>>>>open function successful >>>>>>>>>>>>>>>[compiler-2:14673] orte_rml_base_select: initializing rml >>>>>>>>>>>>>>>component oob >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>30 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>15 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>32 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>33 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>5 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>10 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>12 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>9 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>34 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>2 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>21 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>22 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>45 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>46 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>1 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag >>>>>>>>>>>>>>>27 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>Daemon was launched on node1-128-01 - beginning to initialize >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>WARNING: An invalid value was given for oob_tcp_if_include. This >>>>>>>>>>>>>>>value will be ignored. >>>>>>>>>>>>>>>Local host: node1-128-01 >>>>>>>>>>>>>>>Value: "ib0" >>>>>>>>>>>>>>>Message: Invalid specification (missing "/") >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>None of the TCP networks specified to be included for >>>>>>>>>>>>>>>out-of-band communications >>>>>>>>>>>>>>>could be found: >>>>>>>>>>>>>>>Value given: >>>>>>>>>>>>>>>Please revise the specification and try again. >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>No network interfaces were found for out-of-band communications. >>>>>>>>>>>>>>>We require >>>>>>>>>>>>>>>at least one available network for out-of-band messaging. >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>It looks like orte_init failed for some reason; your parallel >>>>>>>>>>>>>>>process is >>>>>>>>>>>>>>>likely to abort. There are many reasons that a parallel process >>>>>>>>>>>>>>>can >>>>>>>>>>>>>>>fail during orte_init; some of which are due to configuration or >>>>>>>>>>>>>>>environment problems. This failure appears to be an internal >>>>>>>>>>>>>>>failure; >>>>>>>>>>>>>>>here's some additional information (which may only be relevant >>>>>>>>>>>>>>>to an >>>>>>>>>>>>>>>Open MPI developer): >>>>>>>>>>>>>>>orte_oob_base_select failed >>>>>>>>>>>>>>>--> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>srun: error: node1-128-01: task 0: Exited with exit code 213 >>>>>>>>>>>>>>>srun: Terminating job step 661215.0 >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>An ORTE daemon has unexpectedly failed after launch and before >>>>>>>>>>>>>>>communicating back to mpirun. This could be caused by a number >>>>>>>>>>>>>>>of factors, including an inability to create a connection back >>>>>>>>>>>>>>>to mpirun due to a lack of common network interfaces and/or no >>>>>>>>>>>>>>>route found between them. Please check network connectivity >>>>>>>>>>>>>>>(including firewalls and network routing requirements). >>>>>>>>>>>>>>>-------------------------------------------------------------------------- >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] orted_cmd: received halt_vm cmd >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: close: component oob closed >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: close: unloading component oob >>>>>>>>>>>>>>>[compiler-2:14673] [[49095,0],0] TCP SHUTDOWN >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: close: component tcp closed >>>>>>>>>>>>>>>[compiler-2:14673] mca: base: close: unloading component tcp >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>Tue, 12 Aug 2014 18:33:24 +0000 от "Jeff Squyres (jsquyres)" < >>>>>>>>>>>>>>>jsquy...@cisco.com >: >>>>>>>>>>>>>>>>I filed the following ticket: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> https://svn.open-mpi.org/trac/ompi/ticket/4857 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>On Aug 12, 2014, at 12:39 PM, Jeff Squyres (jsquyres) < >>>>>>>>>>>>>>>>jsquy...@cisco.com > wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> (please keep the users list CC'ed) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We talked about this on the weekly engineering call today. >>>>>>>>>>>>>>>>> Ralph has an idea what is happening -- I need to do a little >>>>>>>>>>>>>>>>> investigation today and file a bug. I'll make sure you're >>>>>>>>>>>>>>>>> CC'ed on the bug ticket. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Aug 12, 2014, at 12:27 PM, Timur Ismagilov < >>>>>>>>>>>>>>>>> tismagi...@mail.ru > wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I don't have this error in OMPI 1.9a1r32252 and OMPI 1.8.1 >>>>>>>>>>>>>>>>>> (with --mca oob_tcp_if_include ib0), but in all latest night >>>>>>>>>>>>>>>>>> snapshots i got this error. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Tue, 12 Aug 2014 13:08:12 +0000 от "Jeff Squyres (jsquyres)" >>>>>>>>>>>>>>>>>> < jsquy...@cisco.com >: >>>>>>>>>>>>>>>>>> Are you running any kind of firewall on the node where >>>>>>>>>>>>>>>>>> mpirun is invoked? Open MPI needs to be able to use >>>>>>>>>>>>>>>>>> arbitrary TCP ports between the servers on which it runs. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> This second mail seems to imply a bug in OMPI's >>>>>>>>>>>>>>>>>> oob_tcp_if_include param handling, however -- it's supposed >>>>>>>>>>>>>>>>>> to be able to handle an interface name (not just a network >>>>>>>>>>>>>>>>>> specification). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Ralph -- can you have a look? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Aug 12, 2014, at 8:41 AM, Timur Ismagilov < >>>>>>>>>>>>>>>>>> tismagi...@mail.ru > wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When i add --mca oob_tcp_if_include ib0 (infiniband >>>>>>>>>>>>>>>>>>> interface) to mpirun (as it was here: >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/07/24857.php >>>>>>>>>>>>>>>>>>> ) i got this output: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [isolated] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [isolated] set priority to 0 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [rsh] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [rsh] set priority to 10 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [slurm] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [slurm] set priority to 75 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:base:select:( plm) Selected >>>>>>>>>>>>>>>>>>> component [slurm] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: >>>>>>>>>>>>>>>>>>> registering oob components >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: found >>>>>>>>>>>>>>>>>>> loaded component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: >>>>>>>>>>>>>>>>>>> component tcp register function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: opening oob >>>>>>>>>>>>>>>>>>> components >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: found loaded >>>>>>>>>>>>>>>>>>> component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: component >>>>>>>>>>>>>>>>>>> tcp open function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:oob:select: checking available >>>>>>>>>>>>>>>>>>> component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:oob:select: Querying component [tcp] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] oob:tcp: component_available called >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 1 KERNEL INDEX 1 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 2 KERNEL INDEX 3 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 3 KERNEL INDEX 4 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 4 KERNEL INDEX 5 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 5 KERNEL INDEX 6 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 10.128.0.4 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08792] WORKING INTERFACE 6 KERNEL INDEX 7 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] TCP STARTUP >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] attempting to bind to IPv4 >>>>>>>>>>>>>>>>>>> port 0 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] assigned IPv4 port 53883 >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:oob:select: Adding component to end >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca:oob:select: Found 1 active transports >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: >>>>>>>>>>>>>>>>>>> registering rml components >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: found >>>>>>>>>>>>>>>>>>> loaded component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_register: >>>>>>>>>>>>>>>>>>> component oob has no register or open function >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: opening rml >>>>>>>>>>>>>>>>>>> components >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: found loaded >>>>>>>>>>>>>>>>>>> component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: components_open: component >>>>>>>>>>>>>>>>>>> oob open function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08792] orte_rml_base_select: initializing rml >>>>>>>>>>>>>>>>>>> component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 30 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 15 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 32 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 33 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 5 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 10 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 12 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 9 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 34 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 2 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 21 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 22 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 45 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 46 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 1 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 27 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-128-01 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-128-02 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> WARNING: An invalid value was given for oob_tcp_if_include. >>>>>>>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>>> value will be ignored. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Local host: node1-128-01 >>>>>>>>>>>>>>>>>>> Value: "ib0" >>>>>>>>>>>>>>>>>>> Message: Invalid specification (missing "/") >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> WARNING: An invalid value was given for oob_tcp_if_include. >>>>>>>>>>>>>>>>>>> This >>>>>>>>>>>>>>>>>>> value will be ignored. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Local host: node1-128-02 >>>>>>>>>>>>>>>>>>> Value: "ib0" >>>>>>>>>>>>>>>>>>> Message: Invalid specification (missing "/") >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> None of the TCP networks specified to be included for >>>>>>>>>>>>>>>>>>> out-of-band communications >>>>>>>>>>>>>>>>>>> could be found: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Value given: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please revise the specification and try again. >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> None of the TCP networks specified to be included for >>>>>>>>>>>>>>>>>>> out-of-band communications >>>>>>>>>>>>>>>>>>> could be found: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Value given: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please revise the specification and try again. >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> No network interfaces were found for out-of-band >>>>>>>>>>>>>>>>>>> communications. We require >>>>>>>>>>>>>>>>>>> at least one available network for out-of-band messaging. >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> No network interfaces were found for out-of-band >>>>>>>>>>>>>>>>>>> communications. We require >>>>>>>>>>>>>>>>>>> at least one available network for out-of-band messaging. >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> It looks like orte_init failed for some reason; your >>>>>>>>>>>>>>>>>>> parallel process is >>>>>>>>>>>>>>>>>>> likely to abort. There are many reasons that a parallel >>>>>>>>>>>>>>>>>>> process can >>>>>>>>>>>>>>>>>>> fail during orte_init; some of which are due to >>>>>>>>>>>>>>>>>>> configuration or >>>>>>>>>>>>>>>>>>> environment problems. This failure appears to be an >>>>>>>>>>>>>>>>>>> internal failure; >>>>>>>>>>>>>>>>>>> here's some additional information (which may only be >>>>>>>>>>>>>>>>>>> relevant to an >>>>>>>>>>>>>>>>>>> Open MPI developer): >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> orte_oob_base_select failed >>>>>>>>>>>>>>>>>>> --> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> It looks like orte_init failed for some reason; your >>>>>>>>>>>>>>>>>>> parallel process is >>>>>>>>>>>>>>>>>>> likely to abort. There are many reasons that a parallel >>>>>>>>>>>>>>>>>>> process can >>>>>>>>>>>>>>>>>>> fail during orte_init; some of which are due to >>>>>>>>>>>>>>>>>>> configuration or >>>>>>>>>>>>>>>>>>> environment problems. This failure appears to be an >>>>>>>>>>>>>>>>>>> internal failure; >>>>>>>>>>>>>>>>>>> here's some additional information (which may only be >>>>>>>>>>>>>>>>>>> relevant to an >>>>>>>>>>>>>>>>>>> Open MPI developer): >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> orte_oob_base_select failed >>>>>>>>>>>>>>>>>>> --> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> srun: error: node1-128-02: task 1: Exited with exit code 213 >>>>>>>>>>>>>>>>>>> srun: Terminating job step 657300.0 >>>>>>>>>>>>>>>>>>> srun: error: node1-128-01: task 0: Exited with exit code 213 >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> An ORTE daemon has unexpectedly failed after launch and >>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>> communicating back to mpirun. This could be caused by a >>>>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>>>> of factors, including an inability to create a connection >>>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>> to mpirun due to a lack of common network interfaces and/or >>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>> route found between them. Please check network connectivity >>>>>>>>>>>>>>>>>>> (including firewalls and network routing requirements). >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] orted_cmd: received >>>>>>>>>>>>>>>>>>> halt_vm cmd >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: close: component oob closed >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: close: unloading component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08792] [[42190,0],0] TCP SHUTDOWN >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: close: component tcp closed >>>>>>>>>>>>>>>>>>> [compiler-2:08792] mca: base: close: unloading component tcp >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Tue, 12 Aug 2014 16:14:58 +0400 от Timur Ismagilov < >>>>>>>>>>>>>>>>>>> tismagi...@mail.ru >: >>>>>>>>>>>>>>>>>>> Hello! >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I have Open MPI v1.8.2rc4r32485 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When i run hello_c, I got this error message >>>>>>>>>>>>>>>>>>> $mpirun -np 2 hello_c >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> An ORTE daemon has unexpectedly failed after launch and >>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> communicating back to mpirun. This could be caused by a >>>>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>>>> of factors, including an inability to create a connection >>>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>> to mpirun due to a lack of common network interfaces and/or >>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>> route found between them. Please check network connectivity >>>>>>>>>>>>>>>>>>> (including firewalls and network routing requirements). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> When i run with --debug-daemons --mca plm_base_verbose 5 >>>>>>>>>>>>>>>>>>> -mca oob_base_verbose 10 -mca rml_base_verbose 10 i got >>>>>>>>>>>>>>>>>>> this output: >>>>>>>>>>>>>>>>>>> $mpirun --debug-daemons --mca plm_base_verbose 5 -mca >>>>>>>>>>>>>>>>>>> oob_base_verbose 10 -mca rml_base_verbose 10 -np 2 hello_c >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [isolated] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [isolated] set priority to 0 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [rsh] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [rsh] set priority to 10 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying >>>>>>>>>>>>>>>>>>> component [slurm] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of >>>>>>>>>>>>>>>>>>> component [slurm] set priority to 75 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:base:select:( plm) Selected >>>>>>>>>>>>>>>>>>> component [slurm] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: >>>>>>>>>>>>>>>>>>> registering oob components >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: found >>>>>>>>>>>>>>>>>>> loaded component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: >>>>>>>>>>>>>>>>>>> component tcp register function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: opening oob >>>>>>>>>>>>>>>>>>> components >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: found loaded >>>>>>>>>>>>>>>>>>> component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: component >>>>>>>>>>>>>>>>>>> tcp open function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:oob:select: checking available >>>>>>>>>>>>>>>>>>> component tcp >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:oob:select: Querying component [tcp] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] oob:tcp: component_available called >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 1 KERNEL INDEX 1 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 2 KERNEL INDEX 3 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 10.0.251.53 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 3 KERNEL INDEX 4 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 10.0.0.4 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 4 KERNEL INDEX 5 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 10.2.251.14 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 5 KERNEL INDEX 6 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 10.128.0.4 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08780] WORKING INTERFACE 6 KERNEL INDEX 7 >>>>>>>>>>>>>>>>>>> FAMILY: V4 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding >>>>>>>>>>>>>>>>>>> 93.180.7.38 to our list of V4 connections >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] TCP STARTUP >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] attempting to bind to IPv4 >>>>>>>>>>>>>>>>>>> port 0 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] assigned IPv4 port 38420 >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:oob:select: Adding component to end >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca:oob:select: Found 1 active transports >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: >>>>>>>>>>>>>>>>>>> registering rml components >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: found >>>>>>>>>>>>>>>>>>> loaded component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_register: >>>>>>>>>>>>>>>>>>> component oob has no register or open function >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: opening rml >>>>>>>>>>>>>>>>>>> components >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: found loaded >>>>>>>>>>>>>>>>>>> component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: components_open: component >>>>>>>>>>>>>>>>>>> oob open function successful >>>>>>>>>>>>>>>>>>> [compiler-2:08780] orte_rml_base_select: initializing rml >>>>>>>>>>>>>>>>>>> component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 30 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 15 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 32 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 33 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 5 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 10 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 12 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 9 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 34 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 2 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 21 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 22 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 45 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 46 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 1 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on >>>>>>>>>>>>>>>>>>> tag 27 for peer [[WILDCARD],WILDCARD] >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-08 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-03 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-05 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-02 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-01 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-04 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-07 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon was launched on node1-130-06 - beginning to >>>>>>>>>>>>>>>>>>> initialize >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],3] checking in as pid 7178 on host >>>>>>>>>>>>>>>>>>> node1-130-03 >>>>>>>>>>>>>>>>>>> [node1-130-03:07178] [[42202,0],3] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],2] checking in as pid 13581 on host >>>>>>>>>>>>>>>>>>> node1-130-02 >>>>>>>>>>>>>>>>>>> [node1-130-02:13581] [[42202,0],2] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],1] checking in as pid 17220 on host >>>>>>>>>>>>>>>>>>> node1-130-01 >>>>>>>>>>>>>>>>>>> [node1-130-01:17220] [[42202,0],1] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],5] checking in as pid 6663 on host >>>>>>>>>>>>>>>>>>> node1-130-05 >>>>>>>>>>>>>>>>>>> [node1-130-05:06663] [[42202,0],5] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],8] checking in as pid 6683 on host >>>>>>>>>>>>>>>>>>> node1-130-08 >>>>>>>>>>>>>>>>>>> [node1-130-08:06683] [[42202,0],8] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],7] checking in as pid 7877 on host >>>>>>>>>>>>>>>>>>> node1-130-07 >>>>>>>>>>>>>>>>>>> [node1-130-07:07877] [[42202,0],7] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],4] checking in as pid 7735 on host >>>>>>>>>>>>>>>>>>> node1-130-04 >>>>>>>>>>>>>>>>>>> [node1-130-04:07735] [[42202,0],4] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> Daemon [[42202,0],6] checking in as pid 8451 on host >>>>>>>>>>>>>>>>>>> node1-130-06 >>>>>>>>>>>>>>>>>>> [node1-130-06:08451] [[42202,0],6] orted: up and running - >>>>>>>>>>>>>>>>>>> waiting for commands! >>>>>>>>>>>>>>>>>>> srun: error: node1-130-03: task 2: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> srun: Terminating job step 657040.1 >>>>>>>>>>>>>>>>>>> srun: error: node1-130-02: task 1: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> slurmd[node1-130-04]: *** STEP 657040.1 KILLED AT >>>>>>>>>>>>>>>>>>> 2014-08-12T12:59:07 WITH SIGNAL 9 *** >>>>>>>>>>>>>>>>>>> slurmd[node1-130-07]: *** STEP 657040.1 KILLED AT >>>>>>>>>>>>>>>>>>> 2014-08-12T12:59:07 WITH SIGNAL 9 *** >>>>>>>>>>>>>>>>>>> slurmd[node1-130-06]: *** STEP 657040.1 KILLED AT >>>>>>>>>>>>>>>>>>> 2014-08-12T12:59:07 WITH SIGNAL 9 *** >>>>>>>>>>>>>>>>>>> srun: Job step aborted: Waiting up to 2 seconds for job >>>>>>>>>>>>>>>>>>> step to finish. >>>>>>>>>>>>>>>>>>> srun: error: node1-130-01: task 0: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> srun: error: node1-130-05: task 4: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> srun: error: node1-130-08: task 7: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> srun: error: node1-130-07: task 6: Exited with exit code 1 >>>>>>>>>>>>>>>>>>> srun: error: node1-130-04: task 3: Killed >>>>>>>>>>>>>>>>>>> srun: error: node1-130-06: task 5: Killed >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> An ORTE daemon has unexpectedly failed after launch and >>>>>>>>>>>>>>>>>>> before >>>>>>>>>>>>>>>>>>> communicating back to mpirun. This could be caused by a >>>>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>>>> of factors, including an inability to create a connection >>>>>>>>>>>>>>>>>>> back >>>>>>>>>>>>>>>>>>> to mpirun due to a lack of common network interfaces and/or >>>>>>>>>>>>>>>>>>> no >>>>>>>>>>>>>>>>>>> route found between them. Please check network connectivity >>>>>>>>>>>>>>>>>>> (including firewalls and network routing requirements). >>>>>>>>>>>>>>>>>>> -------------------------------------------------------------------------- >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] orted_cmd: received >>>>>>>>>>>>>>>>>>> halt_vm cmd >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: close: component oob closed >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: close: unloading component oob >>>>>>>>>>>>>>>>>>> [compiler-2:08780] [[42202,0],0] TCP SHUTDOWN >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: close: component tcp closed >>>>>>>>>>>>>>>>>>> [compiler-2:08780] mca: base: close: unloading component tcp >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>> Subscription: >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/08/24987.php >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>> Subscription: >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/08/24988.php >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>> Jeff Squyres >>>>>>>>>>>>>>>>>> jsquy...@cisco.com >>>>>>>>>>>>>>>>>> For corporate legal information go to: >>>>>>>>>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Jeff Squyres >>>>>>>>>>>>>>>>> jsquy...@cisco.com >>>>>>>>>>>>>>>>> For corporate legal information go to: >>>>>>>>>>>>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>> Subscription: >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>> Link to this post: >>>>>>>>>>>>>>>>> http://www.open-mpi.org/community/lists/users/2014/08/25001.php >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>-- >>>>>>>>>>>>>>>>Jeff Squyres >>>>>>>>>>>>>>>>jsquy...@cisco.com >>>>>>>>>>>>>>>>For corporate legal information go to: >>>>>>>>>>>>>>>>http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>>>>users mailing list >>>>>>>>>>>>>>>us...@open-mpi.org >>>>>>>>>>>>>>>Subscription: >>>>>>>>>>>>>>>http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>Link to this post: >>>>>>>>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25086.php >>>>>>>>>>>>>> >>>>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>>>users mailing list >>>>>>>>>>>>>>us...@open-mpi.org >>>>>>>>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>Link to this post: >>>>>>>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25093.php >>>>>>>>>>>>> >>>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>>users mailing list >>>>>>>>>>>>>us...@open-mpi.org >>>>>>>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>Link to this post: >>>>>>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25094.php >>>>>>>>>>>>_______________________________________________ >>>>>>>>>>>>users mailing list >>>>>>>>>>>>us...@open-mpi.org >>>>>>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>Link to this post: >>>>>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25095.php >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>_______________________________________________ >>>>>>>>>>>users mailing list >>>>>>>>>>>us...@open-mpi.org >>>>>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>Link to this post: >>>>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25105.php >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>_______________________________________________ >>>>>>>>>users mailing list >>>>>>>>>us...@open-mpi.org >>>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>Link to this post: >>>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25127.php >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>-- >>>>>>>> >>>>>>>>Kind Regards, >>>>>>>> >>>>>>>>M. _______________________________________________ >>>>>>>>users mailing list >>>>>>>>us...@open-mpi.org >>>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>Link to this post: >>>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25128.php >>>>>>>_______________________________________________ >>>>>>>users mailing list >>>>>>>us...@open-mpi.org >>>>>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>Link to this post: >>>>>>>http://www.open-mpi.org/community/lists/users/2014/08/25129.php >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >>>> >>>>---------------------------------------------------------------------- >>>> >>>> >>>>_______________________________________________ >>>>users mailing list >>>>us...@open-mpi.org >>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>Link to this post: >>>>http://www.open-mpi.org/community/lists/users/2014/08/25154.php >> >> >>
[compiler-2:00750] mca: base: components_register: registering oob components [compiler-2:00750] mca: base: components_register: found loaded component tcp [compiler-2:00750] mca: base: components_register: component tcp register function successful [compiler-2:00750] mca: base: components_open: opening oob components [compiler-2:00750] mca: base: components_open: found loaded component tcp [compiler-2:00750] mca: base: components_open: component tcp open function successful [compiler-2:00750] mca:oob:select: checking available component tcp [compiler-2:00750] mca:oob:select: Querying component [tcp] [compiler-2:00750] oob:tcp: component_available called [compiler-2:00750] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init rejecting loopback interface lo [compiler-2:00750] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init adding 10.0.251.53 to our list of V4 connections [compiler-2:00750] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init adding 10.0.0.4 to our list of V4 connections [compiler-2:00750] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init adding 10.2.251.14 to our list of V4 connections [compiler-2:00750] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init adding 10.128.0.4 to our list of V4 connections [compiler-2:00750] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 [compiler-2:00750] [[33912,0],0] oob:tcp:init adding 93.180.7.38 to our list of V4 connections [compiler-2:00750] [[33912,0],0] TCP STARTUP [compiler-2:00750] [[33912,0],0] attempting to bind to IPv4 port 0 [compiler-2:00750] [[33912,0],0] assigned IPv4 port 37873 [compiler-2:00750] mca:oob:select: Adding component to end [compiler-2:00750] mca:oob:select: Found 1 active transports [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (20, 0) 10.0.128.26:46486 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (21, 11) 10.0.128.27:49851 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (22, 11) 10.0.128.21:45432 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (23, 11) 10.0.128.24:42895 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (24, 11) 10.0.128.25:60899 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (25, 11) 10.0.128.23:46083 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (26, 11) 10.0.128.20:34230 [compiler-2:00750] [[33912,0],0] mca_oob_tcp_listen_thread: new connection: (27, 11) 10.0.128.22:40660 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (20, 2) 10.0.128.26:46486 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.26:46486 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (21, 2) 10.0.128.27:49851 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.27:49851 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (22, 2) 10.0.128.21:45432 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.21:45432 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (23, 2) 10.0.128.24:42895 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.24:42895 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (24, 2) 10.0.128.25:60899 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.25:60899 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (25, 2) 10.0.128.23:46083 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.23:46083 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (26, 2) 10.0.128.20:34230 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.20:34230 [compiler-2:00750] [[33912,0],0] connection_handler: working connection (27, 2) 10.0.128.22:40660 [compiler-2:00750] [[33912,0],0] accept_connection: 10.0.128.22:40660 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 20 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],7] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],7] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],7] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],7] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],7] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],7] in state UNKNOWN on socket 20 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 20 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 20 [compiler-2:00750] [[33912,0],0]-[[33912,0],7] tcp_peer_connected on socket 20 [compiler-2:00750] [[33912,0],0]-[[33912,0],7] accepted: 10.0.0.4 - 10.0.128.26 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 21 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],8] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],8] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],8] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],8] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],8] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],8] in state UNKNOWN on socket 21 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 21 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 21 [compiler-2:00750] [[33912,0],0]-[[33912,0],8] tcp_peer_connected on socket 21 [compiler-2:00750] [[33912,0],0]-[[33912,0],8] accepted: 10.0.0.4 - 10.0.128.27 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 22 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],2] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],2] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],2] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],2] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],2] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],2] in state UNKNOWN on socket 22 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 22 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 22 [compiler-2:00750] [[33912,0],0]-[[33912,0],2] tcp_peer_connected on socket 22 [compiler-2:00750] [[33912,0],0]-[[33912,0],2] accepted: 10.0.0.4 - 10.0.128.21 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:0075[node1-128-20:20264] [[33957,0],1] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-25:14103] [[33957,0],6] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-27:30025] [[33957,0],8] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-23:12872] [[33957,0],4] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-21:13302] [[33957,0],2] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 srun: error: node1-128-20: task 0: Exited with exit code 1 srun: Terminating job step 665096.7 [node1-128-24:00618] [[33957,0],5] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-26:31065] [[33957,0],7] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 [node1-128-22:13413] [[33957,0],3] ORTE_ERROR_LOG: Not found in file runtime/orte_globals.c at line 516 srun: error: node1-128-25: task 5: Exited with exit code 1 srun: error: node1-128-27: task 7: Exited with exit code 1 srun: error: node1-128-23: task 3: Exited with exit code 1 srun: error: node1-128-21: task 1: Exited with exit code 1 srun: error: node1-128-24: task 4: Exited with exit code 1 srun: error: node1-128-22: task 2: Exited with exit code 1 srun: error: node1-128-26: task 6: Exited with exit code 1 handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 24 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],6] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],6] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],6] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],6] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],6] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],6] in state UNKNOWN on socket 24 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 24 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 24 [compiler-2:00750] [[33912,0],0]-[[33912,0],6] tcp_peer_connected on socket 24 [compiler-2:00750] [[33912,0],0]-[[33912,0],6] accepted: 10.0.0.4 - 10.0.128.25 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 25 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],4] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],4] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],4] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],4] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],4] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],4] in state UNKNOWN on socket 25 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 25 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 25 [compiler-2:00750] [[33912,0],0]-[[33912,0],4] tcp_peer_connected on socket 25 [compiler-2:00750] [[33912,0],0]-[[33912,0],4] accepted: 10.0.0.4 - 10.0.128.23 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 26 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],1] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],1] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],1] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],1] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],1] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],1] in state UNKNOWN on socket 26 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 26 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 26 [compiler-2:00750] [[33912,0],0]-[[33912,0],1] tcp_peer_connected on socket 26 [compiler-2:00750] [[33912,0],0]-[[33912,0],1] accepted: 10.0.0.4 - 10.0.128.20 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called [compiler-2:00750] [[33912,0],0] RECV CONNECT ACK FROM UNKNOWN ON SOCKET 27 [compiler-2:00750] [[33912,0],0] waiting for connect ack from UNKNOWN [compiler-2:00750] [[33912,0],0] connect ack received from UNKNOWN [compiler-2:00750] [[33912,0],0] connect-ack recvd from UNKNOWN [compiler-2:00750] [[33912,0],0] mca_oob_tcp_recv_connect: connection from new peer [compiler-2:00750] [[33912,0],0] connect-ack header from [[33912,0],3] is okay [compiler-2:00750] [[33912,0],0] waiting for connect ack from [[33912,0],3] [compiler-2:00750] [[33912,0],0] connect ack received from [[33912,0],3] [compiler-2:00750] [[33912,0],0] connect-ack version from [[33912,0],3] matches ours [compiler-2:00750] [[33912,0],0] connect-ack [[33912,0],3] authenticated [compiler-2:00750] [[33912,0],0] tcp:peer_accept called for peer [[33912,0],3] in state UNKNOWN on socket 27 [compiler-2:00750] [[33912,0],0] SEND CONNECT ACK [compiler-2:00750] [[33912,0],0] send blocking of 46 bytes to socket 27 [compiler-2:00750] [[33912,0],0] blocking send complete to socket 27 [compiler-2:00750] [[33912,0],0]-[[33912,0],3] tcp_peer_connected on socket 27 [compiler-2:00750] [[33912,0],0]-[[33912,0],3] accepted: 10.0.0.4 - 10.0.128.22 nodelay 0 sndbuf 65536 rcvbuf 65536 flags 00000802 [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:set_module called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 14822 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 14822 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.1;tcp://10.0.128.20,10.128.128.20:53976 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],1] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],1] address tcp://10.0.128.20,10.128.128.20:53976 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.20 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.20 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],1] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],1] is listening on net 10.0.128.20 port 53976 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],1] is listening on net 10.128.128.20 port 53976 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 108 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],7] (ORIGIN [[33912,0],7]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.7;tcp://10.0.128.26,10.128.128.26:33562 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],7] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],7] address tcp://10.0.128.26,10.128.128.26:33562 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.26 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.26 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],7] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],7] is listening on net 10.0.128.26 port 33562 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],7] is listening on net 10.128.128.26 port 33562 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],2] (ORIGIN [[33912,0],2]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.2;tcp://10.0.128.21,10.128.128.21:60290 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],2] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],2] address tcp://10.0.128.21,10.128.128.21:60290 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.21 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.21 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],2] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],2] is listening on net 10.0.128.21 port 60290 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],2] is listening on net 10.128.128.21 port 60290 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],8] (ORIGIN [[33912,0],8]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.8;tcp://10.0.128.27,10.128.128.27:56446 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],8] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],8] address tcp://10.0.128.27,10.128.128.27:56446 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.27 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.27 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],8] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],8] is listening on net 10.0.128.27 port 56446 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],8] is listening on net 10.128.128.27 port 56446 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],5] (ORIGIN [[33912,0],5]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.5;tcp://10.0.128.24,10.128.128.24:52008 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],5] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],5] address tcp://10.0.128.24,10.128.128.24:52008 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.24 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.24 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],5] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],5] is listening on net 10.0.128.24 port 52008 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],5] is listening on net 10.128.128.24 port 52008 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],6] (ORIGIN [[33912,0],6]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.6;tcp://10.0.128.25,10.128.128.25:45828 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],6] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],6] address tcp://10.0.128.25,10.128.128.25:45828 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.25 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.25 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],6] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],6] is listening on net 10.0.128.25 port 45828 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],6] is listening on net 10.128.128.25 port 45828 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],4] (ORIGIN [[33912,0],4]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.4;tcp://10.0.128.23,10.128.128.23:53888 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],4] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],4] address tcp://10.0.128.23,10.128.128.23:53888 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.23 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.23 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],4] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],4] is listening on net 10.0.128.23 port 53888 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],4] is listening on net 10.128.128.23 port 53888 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],3] (ORIGIN [[33912,0],3]) OF 108 BYTES FOR DEST [[33912,0],0] TAG 10 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.3;tcp://10.0.128.22,10.128.128.22:56536 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],3] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],3] address tcp://10.0.128.22,10.128.128.22:56536 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.22 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.22 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],3] is reachable via component tcp [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],3] is listening on net 10.0.128.22 port 56536 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],3] is listening on net 10.128.128.22 port 56536 [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.0;tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:37873 [compiler-2:00750] [[33912,0],0]:set_addr peer [[33912,0],0] is me [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.1;tcp://10.0.128.20,10.128.128.20:53976 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],1] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],1] address tcp://10.0.128.20,10.128.128.20:53976 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.20 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.20 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],1] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.2;tcp://10.0.128.21,10.128.128.21:60290 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],2] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],2] address tcp://10.0.128.21,10.128.128.21:60290 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.21 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.21 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],2] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.3;tcp://10.0.128.22,10.128.128.22:56536 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],3] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],3] address tcp://10.0.128.22,10.128.128.22:56536 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.22 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.22 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],3] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.4;tcp://10.0.128.23,10.128.128.23:53888 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],4] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],4] address tcp://10.0.128.23,10.128.128.23:53888 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.23 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.23 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],4] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.5;tcp://10.0.128.24,10.128.128.24:52008 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],5] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],5] address tcp://10.0.128.24,10.128.128.24:52008 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.24 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.24 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],5] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.6;tcp://10.0.128.25,10.128.128.25:45828 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],6] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],6] address tcp://10.0.128.25,10.128.128.25:45828 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.25 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.25 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],6] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.7;tcp://10.0.128.26,10.128.128.26:33562 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],7] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],7] address tcp://10.0.128.26,10.128.128.26:33562 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.26 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.26 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],7] is reachable via component tcp [compiler-2:00750] [[33912,0],0]: set_addr to uri 2222456832.8;tcp://10.0.128.27,10.128.128.27:56446 [compiler-2:00750] [[33912,0],0]:set_addr checking if peer [[33912,0],8] is reachable via component tcp [compiler-2:00750] [[33912,0],0] oob:tcp: working peer [[33912,0],8] address tcp://10.0.128.27,10.128.128.27:56446 [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.0.128.27 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] PASSING ADDR 10.128.128.27 TO MODULE [compiler-2:00750] [[33912,0],0]:tcp set addr for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]: peer [[33912,0],8] is reachable via component tcp [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],1] is listening on net 10.0.128.20 port 53976 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],1] is listening on net 10.128.128.20 port 53976 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],2] is listening on net 10.0.128.21 port 60290 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],2] is listening on net 10.128.128.21 port 60290 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],3] is listening on net 10.0.128.22 port 56536 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],3] is listening on net 10.128.128.22 port 56536 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],4] is listening on net 10.0.128.23 port 53888 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],4] is listening on net 10.128.128.23 port 53888 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],5] is listening on net 10.0.128.24 port 52008 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],5] is listening on net 10.128.128.24 port 52008 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],6] is listening on net 10.0.128.25 port 45828 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],6] is listening on net 10.128.128.25 port 45828 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],7] is listening on net 10.0.128.26 port 33562 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],7] is listening on net 10.128.128.26 port 33562 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],8] is listening on net 10.0.128.27 port 56446 [compiler-2:00750] [[33912,0],0]:tcp:processing set_peer cmd [compiler-2:00750] [[33912,0],0] set_peer: peer [[33912,0],8] is listening on net 10.128.128.27 port 56446 [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],1] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],2] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],3] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],4] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],5] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],6] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],7] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],8] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],1] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],1] OF 1455 BYTES ON SOCKET 26 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],2] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],2] OF 1455 BYTES ON SOCKET 22 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],3] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],3] OF 1455 BYTES ON SOCKET 27 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],4] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],4] OF 1455 BYTES ON SOCKET 25 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],5] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],5] OF 1455 BYTES ON SOCKET 23 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],6] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],6] OF 1455 BYTES ON SOCKET 24 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],7] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],7] OF 1455 BYTES ON SOCKET 20 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],8] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],8] OF 1455 BYTES ON SOCKET 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 54 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],2] (ORIGIN [[33912,0],2]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],4] (ORIGIN [[33912,0],4]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],5] (ORIGIN [[33912,0],5]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],6] (ORIGIN [[33912,0],6]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],7] (ORIGIN [[33912,0],7]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],3] (ORIGIN [[33912,0],3]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],8] (ORIGIN [[33912,0],8]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 54 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 21 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 21 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 181 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 181 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [warn] Epoll ADD(4) on fd 2 failed. Old events were 0; read change was 0 (none); write change was 1 (add): Operation not permitted [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 117 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 117 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 389 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 389 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 78 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 78 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 88 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 88 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 478 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 478 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 90 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 90 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 142 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 142 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 84 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 84 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 124 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 124 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 127 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 127 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 351 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 351 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 481 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 481 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 130 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 130 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 390 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 390 BYTES FOR DEST [[33912,0],0] TAG 33 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],1] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],2] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],3] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],4] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],5] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],6] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],7] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],8] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],1] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],1] OF 448 BYTES ON SOCKET 26 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],2] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],2] OF 448 BYTES ON SOCKET 22 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],3] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],3] OF 448 BYTES ON SOCKET 27 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],4] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],4] OF 448 BYTES ON SOCKET 25 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],5] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],5] OF 448 BYTES ON SOCKET 23 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],6] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],6] OF 448 BYTES ON SOCKET 24 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],7] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],7] OF 448 BYTES ON SOCKET 20 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],8] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],8] OF 448 BYTES ON SOCKET 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 29 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 29 BYTES FOR DEST [[33912,0],0] TAG 33 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],1] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],2] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],3] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],4] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],5] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],6] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],7] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],8] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],1] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],1] OF 87 BYTES ON SOCKET 26 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],2] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],2] OF 87 BYTES ON SOCKET 22 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],3] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],3] OF 87 BYTES ON SOCKET 27 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],4] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],4] OF 87 BYTES ON SOCKET 25 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],5] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],5] OF 87 BYTES ON SOCKET 23 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],6] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],6] OF 87 BYTES ON SOCKET 24 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],7] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],7] OF 87 BYTES ON SOCKET 20 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],8] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],8] OF 87 BYTES ON SOCKET 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 29 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 29 BYTES FOR DEST [[33912,0],0] TAG 33 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],1] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],2] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],3] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],4] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],5] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],6] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],7] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],8] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 201 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 201 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],1] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],1] OF 87 BYTES ON SOCKET 26 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],2] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],2] OF 87 BYTES ON SOCKET 22 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],3] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],3] OF 87 BYTES ON SOCKET 27 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],4] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],4] OF 87 BYTES ON SOCKET 25 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],5] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],5] OF 87 BYTES ON SOCKET 23 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],6] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],6] OF 87 BYTES ON SOCKET 24 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],7] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],7] OF 87 BYTES ON SOCKET 20 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],8] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],8] OF 87 BYTES ON SOCKET 21 [warn] Epoll ADD(4) on fd 1 failed. Old events were 0; read change was 0 (none); write change was 1 (add): Operation not permitted [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 69 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 69 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 212 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 212 BYTES FOR DEST [[33912,0],0] TAG 2 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate data region of size 54 [compiler-2:00750] [[33912,0],0] RECVD COMPLETE MESSAGE FROM [[33912,0],1] (ORIGIN [[33912,0],1]) OF 54 BYTES FOR DEST [[33912,0],0] TAG 5 [compiler-2:00750] [[33912,0],0] DELIVERING TO RML [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] OOB_SEND: rml_oob_send.c:199 [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],1] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],2] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],3] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],4] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],5] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],6] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],7] [compiler-2:00750] [[33912,0],0] oob:base:send to target [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:base:send known transport for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] oob:tcp:send_nb to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:478] post send to [[33912,0],8] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],1]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],1] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],1] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],2]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],2] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],2] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],3]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],3] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],3] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],4]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],4] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],4] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],5]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],5] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],5] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],6]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],6] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],6] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],7]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],7] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],7] [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:415] processing send to peer [[33912,0],8]:15 [compiler-2:00750] [[33912,0],0] tcp:send_nb: already connected to [[33912,0],8] - queueing for send [compiler-2:00750] [[33912,0],0]:[oob_tcp.c:442] queue send to [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],1] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],1] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],1] OF 30 BYTES ON SOCKET 26 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],2] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],2] OF 30 BYTES ON SOCKET 22 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],3] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],3] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],3] OF 30 BYTES ON SOCKET 27 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],4] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],4] OF 30 BYTES ON SOCKET 25 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],5] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],5] OF 30 BYTES ON SOCKET 23 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],6] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],6] OF 30 BYTES ON SOCKET 24 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],7] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],7] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],7] OF 30 BYTES ON SOCKET 20 [compiler-2:00750] [[33912,0],0] tcp:send_handler called to send to peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:send_handler SENDING TO [[33912,0],8] [compiler-2:00750] [[33912,0],0] MESSAGE SEND COMPLETE TO [[33912,0],8] OF 30 BYTES ON SOCKET 21 [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],2] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],2] sd 22 state CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],3] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],3] sd 27 state CONNECTED [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],2] [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],3] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],4] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],4] sd 25 state CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],5] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],5] sd 23 state CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],6] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],6] sd 24 state CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],8] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],8] sd 21 state CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],7] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],7] sd 20 state CONNECTED [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],4] [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],5] [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],6] [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],8] [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],7] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler called for peer [[33912,0],1] [compiler-2:00750] [[33912,0],0]:tcp:recv:handler CONNECTED [compiler-2:00750] [[33912,0],0]:tcp:recv:handler allocate new recv msg [compiler-2:00750] [[33912,0],0]:tcp:recv:handler read hdr [compiler-2:00750] [[33912,0],0]-[[33912,0],1] mca_oob_tcp_msg_recv: peer closed connection [compiler-2:00750] [[33912,0],0] tcp_peer_close for [[33912,0],1] sd 26 state CONNECTED [compiler-2:00750] [[33912,0],0] tcp:lost connection called for peer [[33912,0],1] Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI semenov@compiler-2 Distribution, ident: 1.9a1r32570, repo rev: r32570, Aug 21, 2014 (nightly snapshot tarball), 146) [node1-128-20:20405] mca: base: components_register: registering oob components [node1-128-20:20405] mca: base: components_register: found loaded component tcp [node1-128-20:20405] mca: base: components_register: component tcp register function successful [node1-128-20:20405] mca: base: components_open: opening oob components [node1-128-20:20405] mca: base: components_open: found loaded component tcp [node1-128-20:20405] mca: base: components_open: component tcp open function successful [node1-128-20:20405] mca:oob:select: checking available component tcp [node1-128-20:20405] mca:oob:select: Querying component [tcp] [node1-128-20:20405] oob:tcp: component_available called [node1-128-20:20405] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [node1-128-20:20405] [[33912,1],0] oob:tcp:init rejecting loopback interface lo [node1-128-20:20405] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [node1-128-20:20405] [[33912,1],0] oob:tcp:init adding 10.0.128.20 to our list of V4 connections [node1-128-20:20405] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [node1-128-20:20405] [[33912,1],0] oob:tcp:init adding 10.128.128.20 to our list of V4 connections [node1-128-20:20405] [[33912,1],0] TCP STARTUP [node1-128-20:20405] [[33912,1],0] attempting to bind to IPv4 port 0 [node1-128-20:20405] [[33912,1],0] assigned IPv4 port 41140 [node1-128-20:20405] mca:oob:select: Adding component to end [node1-128-20:20405] mca:oob:select: Found 1 active transports [node1-128-20:20405] [[33912,1],0]: set_addr to uri 2222456832.1;tcp://10.0.128.20,10.128.128.20:53976 [node1-128-20:20405] [[33912,1],0]:set_addr checking if peer [[33912,0],1] is reachable via component tcp [node1-128-20:20405] [[33912,1],0] oob:tcp: working peer [[33912,0],1] address tcp://10.0.128.20,10.128.128.20:53976 [node1-128-20:20405] [[33912,1],0] PASSING ADDR 10.0.128.20 TO MODULE [node1-128-20:20405] [[33912,1],0]:tcp set addr for peer [[33912,0],1] [node1-128-20:20405] [[33912,1],0] PASSING ADDR 10.128.128.20 TO MODULE [node1-128-20:20405] [[33912,1],0]:tcp set addr for peer [[33912,0],1] [node1-128-20:20405] [[33912,1],0]: peer [[33912,0],1] is reachable via component tcp [node1-128-20:20405] [[33912,1],0]:tcp:processing set_peer cmd [node1-128-20:20405] [[33912,1],0] SET_PEER ADDING PEER [[33912,0],1] [node1-128-20:20405] [[33912,1],0] set_peer: peer [[33912,0],1] is listening on net 10.0.128.20 port 53976 [node1-128-20:20405] [[33912,1],0]:tcp:processing set_peer cmd [node1-128-20:20405] [[33912,1],0] set_peer: peer [[33912,0],1] is listening on net 10.128.128.20 port 53976 [node1-128-20:20405] [[33912,1],0] TCP SHUTDOWN [node1-128-20:20405] [[33912,1],0] RELEASING PEER OBJ [[33912,0],1] [node1-128-20:20405] mca: base: close: component tcp closed [node1-128-20:20405] mca: base: close: unloading component tcp [compiler-2:00750] [[33912,0],0] TCP SHUTDOWN [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],8] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],2] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],7] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],1] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],6] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],5] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],4] [compiler-2:00750] [[33912,0],0] RELEASING PEER OBJ [[33912,0],3] [compiler-2:00750] mca: base: close: component tcp closed [compiler-2:00750] mca: base: close: unloading component tcp
[compiler-2:00793] mca: base: components_register: registering oob components [compiler-2:00793] mca: base: components_register: found loaded component tcp [compiler-2:00793] mca: base: components_register: component tcp register function successful [compiler-2:00793] mca: base: components_open: opening oob components [compiler-2:00793] mca: base: components_open: found loaded component tcp [compiler-2:00793] mca: base: components_open: component tcp open function successful [compiler-2:00793] mca:oob:select: checking available component tcp [compiler-2:00793] mca:oob:select: Querying component [tcp] [compiler-2:00793] oob:tcp: component_available called [compiler-2:00793] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init rejecting interface lo (not in include list) [compiler-2:00793] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init rejecting interface eth1 (not in include list) [compiler-2:00793] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init rejecting interface eth2 (not in include list) [compiler-2:00793] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init rejecting interface eth3 (not in include list) [compiler-2:00793] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init adding 10.128.0.4 to our list of V4 connections [compiler-2:00793] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 [compiler-2:00793] [[34191,0],0] oob:tcp:init rejecting interface eth2.911 (not in include list) [compiler-2:00793] [[34191,0],0] TCP STARTUP [compiler-2:00793] [[34191,0],0] attempting to bind to IPv4 port 0 [compiler-2:00793] [[34191,0],0] assigned IPv4 port 46598 [compiler-2:00793] mca:oob:select: Adding component to end [compiler-2:00793] mca:oob:select: Found 1 active transports -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-26 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-27 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- srun: error: node1-128-26: task 6: Exited with exit code 213 srun: Terminating job step 665096.12 -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-21 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-20 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-22 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- None of the TCP networks specified to be included for out-of-band communications could be found: Value given: Please revise the specification and try again. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No network interfaces were found for out-of-band communications. We require at least one available network for out-of-band messaging. -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_oob_base_select failed --> Returned value (null) (-43) instead of ORTE_SUCCESS -------------------------------------------------------------------------- srun: error: node1-128-27: task 7: Exited with exit code 213 -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This srun: error: node1-128-21: task 1: Exited with exit code 213 value will be ignored. Local host: node1-128-24 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- None of the TCP networks specified to be included for out-of-band communications could be found: Value given: Please revise the specification and try again. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No network interfaces were found for out-of-band communications. We require at least one available network for out-of-band messaging. -------------------------------------------------------------------------- -------------------------------------------------------------------------- WARNING: An invalid value was given for oob_tcp_if_include. This value will be ignored. Local host: node1-128-25 Value: "ib0" Message: Invalid specification (missing "/") -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_oob_base_select failed --> Returned value (null) (-43) instead of ORTE_SUCCESS -------------------------------------------------------------------------- -------------------------------------------------------------------------- None of the TCP networks specified to be included for out-of-band communications could be found: Value given: Please revise the specification and try again. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No network interfaces were found for out-of-band communications. We require at least one available network for out-of-band messaging. -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_oob_base_select failed --> Returned value (null) (-43) instead of ORTE_SUCCESS -------------------------------------------------------------------------- slurmd[node1-128-23]: *** STEP 665096.12 KILLED AT 2014-08-28T12:51:42 WITH SIGNAL 9 *** -------------------------------------------------------------------------- None of the TCP networks specified to be included for out-of-band communications could be found: Value given: Please revise the specification and try again. -------------------------------------------------------------------------- -------------------------------------------------------------------------- No network interfaces were found for out-of-band communications. We require at least one available network for out-of-band messaging. -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): orte_oob_base_select failed --> Returned value (null) (-43) instead of ORTE_SUCCESS -------------------------------------------------------------------------- srun: Job step aborted: Waiting up to 2 seconds for job step to finish. srun: error: node1-128-20: task 0: Exited with exit code 213 srun: error: node1-128-22: task 2: Exited with exit code 213 srun: error: node1-128-24: task 4: Killed srun: error: node1-128-23: task 3: Killed srun: error: node1-128-25: task 5: Exited with exit code 213 -------------------------------------------------------------------------- An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This could be caused by a number of factors, including an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). -------------------------------------------------------------------------- [compiler-2:00793] [[34191,0],0] TCP SHUTDOWN [compiler-2:00793] mca: base: close: component tcp closed [compiler-2:00793] mca: base: close: unloading component tcp
