Have i I any opportunity to run mpi jobs?
Wed, 20 Aug 2014 10:48:38 -0700 от Ralph Castain <r...@open-mpi.org>: >yes, i know - it is cmr'd > >On Aug 20, 2014, at 10:26 AM, Mike Dubman < mi...@dev.mellanox.co.il > wrote: >>btw, we get same error in v1.8 branch as well. >> >> >>On Wed, Aug 20, 2014 at 8:06 PM, Ralph Castain < r...@open-mpi.org > wrote: >>>It was not yet fixed - but should be now. >>> >>>On Aug 20, 2014, at 6:39 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>>Hello! >>>> >>>>As i can see, the bug is fixed, but in Open MPI v1.9a1r32516 i still have >>>>the problem >>>> >>>>a) >>>>$ mpirun -np 1 ./hello_c >>>>-------------------------------------------------------------------------- >>>>An ORTE daemon has unexpectedly failed after launch and before >>>>communicating back to mpirun. This could be caused by a number >>>>of factors, including an inability to create a connection back >>>>to mpirun due to a lack of common network interfaces and/or no >>>>route found between them. Please check network connectivity >>>>(including firewalls and network routing requirements). >>>>-------------------------------------------------------------------------- >>>>b) >>>>$ mpirun --mca oob_tcp_if_include ib0 -np 1 ./hello_c >>>>-------------------------------------------------------------------------- >>>>An ORTE daemon has unexpectedly failed after launch and before >>>>communicating back to mpirun. This could be caused by a number >>>>of factors, including an inability to create a connection back >>>>to mpirun due to a lack of common network interfaces and/or no >>>>route found between them. Please check network connectivity >>>>(including firewalls and network routing requirements). >>>>-------------------------------------------------------------------------- >>>> >>>>c) >>>> >>>>$ mpirun --mca oob_tcp_if_include ib0 -debug-daemons --mca plm_base_verbose >>>>5 -mca oob_base_verbose 10 -mca rml_base_verbose 10 -np 1 ./hello_c >>>>[compiler-2:14673] mca:base:select:( plm) Querying component [isolated] >>>>[compiler-2:14673] mca:base:select:( plm) Query of component [isolated] set >>>>priority to 0 >>>>[compiler-2:14673] mca:base:select:( plm) Querying component [rsh] >>>>[compiler-2:14673] mca:base:select:( plm) Query of component [rsh] set >>>>priority to 10 >>>>[compiler-2:14673] mca:base:select:( plm) Querying component [slurm] >>>>[compiler-2:14673] mca:base:select:( plm) Query of component [slurm] set >>>>priority to 75 >>>>[compiler-2:14673] mca:base:select:( plm) Selected component [slurm] >>>>[compiler-2:14673] mca: base: components_register: registering oob >>>>components >>>>[compiler-2:14673] mca: base: components_register: found loaded component >>>>tcp >>>>[compiler-2:14673] mca: base: components_register: component tcp register >>>>function successful >>>>[compiler-2:14673] mca: base: components_open: opening oob components >>>>[compiler-2:14673] mca: base: components_open: found loaded component tcp >>>>[compiler-2:14673] mca: base: components_open: component tcp open function >>>>successful >>>>[compiler-2:14673] mca:oob:select: checking available component tcp >>>>[compiler-2:14673] mca:oob:select: Querying component [tcp] >>>>[compiler-2:14673] oob:tcp: component_available called >>>>[compiler-2:14673] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>>>[compiler-2:14673] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>>>[compiler-2:14673] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>>>[compiler-2:14673] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>>>[compiler-2:14673] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>>>[compiler-2:14673] [[49095,0],0] oob:tcp:init adding 10.128.0.4 to our list >>>>of V4 connections >>>>[compiler-2:14673] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>>>[compiler-2:14673] [[49095,0],0] TCP STARTUP >>>>[compiler-2:14673] [[49095,0],0] attempting to bind to IPv4 port 0 >>>>[compiler-2:14673] [[49095,0],0] assigned IPv4 port 59460 >>>>[compiler-2:14673] mca:oob:select: Adding component to end >>>>[compiler-2:14673] mca:oob:select: Found 1 active transports >>>>[compiler-2:14673] mca: base: components_register: registering rml >>>>components >>>>[compiler-2:14673] mca: base: components_register: found loaded component >>>>oob >>>>[compiler-2:14673] mca: base: components_register: component oob has no >>>>register or open function >>>>[compiler-2:14673] mca: base: components_open: opening rml components >>>>[compiler-2:14673] mca: base: components_open: found loaded component oob >>>>[compiler-2:14673] mca: base: components_open: component oob open function >>>>successful >>>>[compiler-2:14673] orte_rml_base_select: initializing rml component oob >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 30 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 15 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 32 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 33 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 5 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 10 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 12 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 9 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 34 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 2 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 21 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 22 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 45 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 46 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 1 for peer >>>>[[WILDCARD],WILDCARD] >>>>[compiler-2:14673] [[49095,0],0] posting recv >>>>[compiler-2:14673] [[49095,0],0] posting persistent recv on tag 27 for peer >>>>[[WILDCARD],WILDCARD] >>>>Daemon was launched on node1-128-01 - beginning to initialize >>>>-------------------------------------------------------------------------- >>>>WARNING: An invalid value was given for oob_tcp_if_include. This >>>>value will be ignored. >>>>Local host: node1-128-01 >>>>Value: "ib0" >>>>Message: Invalid specification (missing "/") >>>>-------------------------------------------------------------------------- >>>>-------------------------------------------------------------------------- >>>>None of the TCP networks specified to be included for out-of-band >>>>communications >>>>could be found: >>>>Value given: >>>>Please revise the specification and try again. >>>>-------------------------------------------------------------------------- >>>>-------------------------------------------------------------------------- >>>>No network interfaces were found for out-of-band communications. We require >>>>at least one available network for out-of-band messaging. >>>>-------------------------------------------------------------------------- >>>>-------------------------------------------------------------------------- >>>>It looks like orte_init failed for some reason; your parallel process is >>>>likely to abort. There are many reasons that a parallel process can >>>>fail during orte_init; some of which are due to configuration or >>>>environment problems. This failure appears to be an internal failure; >>>>here's some additional information (which may only be relevant to an >>>>Open MPI developer): >>>>orte_oob_base_select failed >>>>--> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>-------------------------------------------------------------------------- >>>>srun: error: node1-128-01: task 0: Exited with exit code 213 >>>>srun: Terminating job step 661215.0 >>>>-------------------------------------------------------------------------- >>>>An ORTE daemon has unexpectedly failed after launch and before >>>>communicating back to mpirun. This could be caused by a number >>>>of factors, including an inability to create a connection back >>>>to mpirun due to a lack of common network interfaces and/or no >>>>route found between them. Please check network connectivity >>>>(including firewalls and network routing requirements). >>>>-------------------------------------------------------------------------- >>>>[compiler-2:14673] [[49095,0],0] orted_cmd: received halt_vm cmd >>>>[compiler-2:14673] mca: base: close: component oob closed >>>>[compiler-2:14673] mca: base: close: unloading component oob >>>>[compiler-2:14673] [[49095,0],0] TCP SHUTDOWN >>>>[compiler-2:14673] mca: base: close: component tcp closed >>>>[compiler-2:14673] mca: base: close: unloading component tcp >>>> >>>> >>>>Tue, 12 Aug 2014 18:33:24 +0000 от "Jeff Squyres (jsquyres)" < >>>>jsquy...@cisco.com >: >>>>>I filed the following ticket: >>>>> >>>>> https://svn.open-mpi.org/trac/ompi/ticket/4857 >>>>> >>>>> >>>>>On Aug 12, 2014, at 12:39 PM, Jeff Squyres (jsquyres) < jsquy...@cisco.com >>>>>> wrote: >>>>> >>>>>> (please keep the users list CC'ed) >>>>>> >>>>>> We talked about this on the weekly engineering call today. Ralph has an >>>>>> idea what is happening -- I need to do a little investigation today and >>>>>> file a bug. I'll make sure you're CC'ed on the bug ticket. >>>>>> >>>>>> >>>>>> >>>>>> On Aug 12, 2014, at 12:27 PM, Timur Ismagilov < tismagi...@mail.ru > >>>>>> wrote: >>>>>> >>>>>>> I don't have this error in OMPI 1.9a1r32252 and OMPI 1.8.1 (with --mca >>>>>>> oob_tcp_if_include ib0), but in all latest night snapshots i got this >>>>>>> error. >>>>>>> >>>>>>> >>>>>>> Tue, 12 Aug 2014 13:08:12 +0000 от "Jeff Squyres (jsquyres)" < >>>>>>> jsquy...@cisco.com >: >>>>>>> Are you running any kind of firewall on the node where mpirun is >>>>>>> invoked? Open MPI needs to be able to use arbitrary TCP ports between >>>>>>> the servers on which it runs. >>>>>>> >>>>>>> This second mail seems to imply a bug in OMPI's oob_tcp_if_include >>>>>>> param handling, however -- it's supposed to be able to handle an >>>>>>> interface name (not just a network specification). >>>>>>> >>>>>>> Ralph -- can you have a look? >>>>>>> >>>>>>> >>>>>>> On Aug 12, 2014, at 8:41 AM, Timur Ismagilov < tismagi...@mail.ru > >>>>>>> wrote: >>>>>>> >>>>>>>> When i add --mca oob_tcp_if_include ib0 (infiniband interface) to >>>>>>>> mpirun (as it was here: >>>>>>>> http://www.open-mpi.org/community/lists/users/2014/07/24857.php ) i >>>>>>>> got this output: >>>>>>>> >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying component [isolated] >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of component >>>>>>>> [isolated] set priority to 0 >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying component [rsh] >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of component [rsh] set >>>>>>>> priority to 10 >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Querying component [slurm] >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Query of component [slurm] >>>>>>>> set priority to 75 >>>>>>>> [compiler-2:08792] mca:base:select:( plm) Selected component [slurm] >>>>>>>> [compiler-2:08792] mca: base: components_register: registering oob >>>>>>>> components >>>>>>>> [compiler-2:08792] mca: base: components_register: found loaded >>>>>>>> component tcp >>>>>>>> [compiler-2:08792] mca: base: components_register: component tcp >>>>>>>> register function successful >>>>>>>> [compiler-2:08792] mca: base: components_open: opening oob components >>>>>>>> [compiler-2:08792] mca: base: components_open: found loaded component >>>>>>>> tcp >>>>>>>> [compiler-2:08792] mca: base: components_open: component tcp open >>>>>>>> function successful >>>>>>>> [compiler-2:08792] mca:oob:select: checking available component tcp >>>>>>>> [compiler-2:08792] mca:oob:select: Querying component [tcp] >>>>>>>> [compiler-2:08792] oob:tcp: component_available called >>>>>>>> [compiler-2:08792] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>>>>>>> [compiler-2:08792] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>>>>>>> [compiler-2:08792] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>>>>>>> [compiler-2:08792] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>>>>>>> [compiler-2:08792] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>>>>>>> [compiler-2:08792] [[42190,0],0] oob:tcp:init adding 10.128.0.4 to our >>>>>>>> list of V4 connections >>>>>>>> [compiler-2:08792] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>>>>>>> [compiler-2:08792] [[42190,0],0] TCP STARTUP >>>>>>>> [compiler-2:08792] [[42190,0],0] attempting to bind to IPv4 port 0 >>>>>>>> [compiler-2:08792] [[42190,0],0] assigned IPv4 port 53883 >>>>>>>> [compiler-2:08792] mca:oob:select: Adding component to end >>>>>>>> [compiler-2:08792] mca:oob:select: Found 1 active transports >>>>>>>> [compiler-2:08792] mca: base: components_register: registering rml >>>>>>>> components >>>>>>>> [compiler-2:08792] mca: base: components_register: found loaded >>>>>>>> component oob >>>>>>>> [compiler-2:08792] mca: base: components_register: component oob has >>>>>>>> no register or open function >>>>>>>> [compiler-2:08792] mca: base: components_open: opening rml components >>>>>>>> [compiler-2:08792] mca: base: components_open: found loaded component >>>>>>>> oob >>>>>>>> [compiler-2:08792] mca: base: components_open: component oob open >>>>>>>> function successful >>>>>>>> [compiler-2:08792] orte_rml_base_select: initializing rml component oob >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 30 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 15 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 32 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 33 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 5 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 10 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 12 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 9 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 34 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 2 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 21 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 22 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 45 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 46 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 1 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08792] [[42190,0],0] posting recv >>>>>>>> [compiler-2:08792] [[42190,0],0] posting persistent recv on tag 27 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> Daemon was launched on node1-128-01 - beginning to initialize >>>>>>>> Daemon was launched on node1-128-02 - beginning to initialize >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> WARNING: An invalid value was given for oob_tcp_if_include. This >>>>>>>> value will be ignored. >>>>>>>> >>>>>>>> Local host: node1-128-01 >>>>>>>> Value: "ib0" >>>>>>>> Message: Invalid specification (missing "/") >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> WARNING: An invalid value was given for oob_tcp_if_include. This >>>>>>>> value will be ignored. >>>>>>>> >>>>>>>> Local host: node1-128-02 >>>>>>>> Value: "ib0" >>>>>>>> Message: Invalid specification (missing "/") >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> None of the TCP networks specified to be included for out-of-band >>>>>>>> communications >>>>>>>> could be found: >>>>>>>> >>>>>>>> Value given: >>>>>>>> >>>>>>>> Please revise the specification and try again. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> None of the TCP networks specified to be included for out-of-band >>>>>>>> communications >>>>>>>> could be found: >>>>>>>> >>>>>>>> Value given: >>>>>>>> >>>>>>>> Please revise the specification and try again. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> No network interfaces were found for out-of-band communications. We >>>>>>>> require >>>>>>>> at least one available network for out-of-band messaging. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> No network interfaces were found for out-of-band communications. We >>>>>>>> require >>>>>>>> at least one available network for out-of-band messaging. >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> It looks like orte_init failed for some reason; your parallel process >>>>>>>> is >>>>>>>> likely to abort. There are many reasons that a parallel process can >>>>>>>> fail during orte_init; some of which are due to configuration or >>>>>>>> environment problems. This failure appears to be an internal failure; >>>>>>>> here's some additional information (which may only be relevant to an >>>>>>>> Open MPI developer): >>>>>>>> >>>>>>>> orte_oob_base_select failed >>>>>>>> --> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> It looks like orte_init failed for some reason; your parallel process >>>>>>>> is >>>>>>>> likely to abort. There are many reasons that a parallel process can >>>>>>>> fail during orte_init; some of which are due to configuration or >>>>>>>> environment problems. This failure appears to be an internal failure; >>>>>>>> here's some additional information (which may only be relevant to an >>>>>>>> Open MPI developer): >>>>>>>> >>>>>>>> orte_oob_base_select failed >>>>>>>> --> Returned value (null) (-43) instead of ORTE_SUCCESS >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> srun: error: node1-128-02: task 1: Exited with exit code 213 >>>>>>>> srun: Terminating job step 657300.0 >>>>>>>> srun: error: node1-128-01: task 0: Exited with exit code 213 >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> An ORTE daemon has unexpectedly failed after launch and before >>>>>>>> communicating back to mpirun. This could be caused by a number >>>>>>>> of factors, including an inability to create a connection back >>>>>>>> to mpirun due to a lack of common network interfaces and/or no >>>>>>>> route found between them. Please check network connectivity >>>>>>>> (including firewalls and network routing requirements). >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> [compiler-2:08792] [[42190,0],0] orted_cmd: received halt_vm cmd >>>>>>>> [compiler-2:08792] mca: base: close: component oob closed >>>>>>>> [compiler-2:08792] mca: base: close: unloading component oob >>>>>>>> [compiler-2:08792] [[42190,0],0] TCP SHUTDOWN >>>>>>>> [compiler-2:08792] mca: base: close: component tcp closed >>>>>>>> [compiler-2:08792] mca: base: close: unloading component tcp >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Tue, 12 Aug 2014 16:14:58 +0400 от Timur Ismagilov < >>>>>>>> tismagi...@mail.ru >: >>>>>>>> Hello! >>>>>>>> >>>>>>>> I have Open MPI v1.8.2rc4r32485 >>>>>>>> >>>>>>>> When i run hello_c, I got this error message >>>>>>>> $mpirun -np 2 hello_c >>>>>>>> >>>>>>>> An ORTE daemon has unexpectedly failed after launch and before >>>>>>>> >>>>>>>> communicating back to mpirun. This could be caused by a number >>>>>>>> of factors, including an inability to create a connection back >>>>>>>> to mpirun due to a lack of common network interfaces and/or no >>>>>>>> route found between them. Please check network connectivity >>>>>>>> (including firewalls and network routing requirements). >>>>>>>> >>>>>>>> When i run with --debug-daemons --mca plm_base_verbose 5 -mca >>>>>>>> oob_base_verbose 10 -mca rml_base_verbose 10 i got this output: >>>>>>>> $mpirun --debug-daemons --mca plm_base_verbose 5 -mca oob_base_verbose >>>>>>>> 10 -mca rml_base_verbose 10 -np 2 hello_c >>>>>>>> >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying component [isolated] >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of component >>>>>>>> [isolated] set priority to 0 >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying component [rsh] >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of component [rsh] set >>>>>>>> priority to 10 >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Querying component [slurm] >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Query of component [slurm] >>>>>>>> set priority to 75 >>>>>>>> [compiler-2:08780] mca:base:select:( plm) Selected component [slurm] >>>>>>>> [compiler-2:08780] mca: base: components_register: registering oob >>>>>>>> components >>>>>>>> [compiler-2:08780] mca: base: components_register: found loaded >>>>>>>> component tcp >>>>>>>> [compiler-2:08780] mca: base: components_register: component tcp >>>>>>>> register function successful >>>>>>>> [compiler-2:08780] mca: base: components_open: opening oob components >>>>>>>> [compiler-2:08780] mca: base: components_open: found loaded component >>>>>>>> tcp >>>>>>>> [compiler-2:08780] mca: base: components_open: component tcp open >>>>>>>> function successful >>>>>>>> [compiler-2:08780] mca:oob:select: checking available component tcp >>>>>>>> [compiler-2:08780] mca:oob:select: Querying component [tcp] >>>>>>>> [compiler-2:08780] oob:tcp: component_available called >>>>>>>> [compiler-2:08780] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>>>>>>> [compiler-2:08780] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding 10.0.251.53 to >>>>>>>> our list of V4 connections >>>>>>>> [compiler-2:08780] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding 10.0.0.4 to our >>>>>>>> list of V4 connections >>>>>>>> [compiler-2:08780] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding 10.2.251.14 to >>>>>>>> our list of V4 connections >>>>>>>> [compiler-2:08780] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding 10.128.0.4 to our >>>>>>>> list of V4 connections >>>>>>>> [compiler-2:08780] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>>>>>>> [compiler-2:08780] [[42202,0],0] oob:tcp:init adding 93.180.7.38 to >>>>>>>> our list of V4 connections >>>>>>>> [compiler-2:08780] [[42202,0],0] TCP STARTUP >>>>>>>> [compiler-2:08780] [[42202,0],0] attempting to bind to IPv4 port 0 >>>>>>>> [compiler-2:08780] [[42202,0],0] assigned IPv4 port 38420 >>>>>>>> [compiler-2:08780] mca:oob:select: Adding component to end >>>>>>>> [compiler-2:08780] mca:oob:select: Found 1 active transports >>>>>>>> [compiler-2:08780] mca: base: components_register: registering rml >>>>>>>> components >>>>>>>> [compiler-2:08780] mca: base: components_register: found loaded >>>>>>>> component oob >>>>>>>> [compiler-2:08780] mca: base: components_register: component oob has >>>>>>>> no register or open function >>>>>>>> [compiler-2:08780] mca: base: components_open: opening rml components >>>>>>>> [compiler-2:08780] mca: base: components_open: found loaded component >>>>>>>> oob >>>>>>>> [compiler-2:08780] mca: base: components_open: component oob open >>>>>>>> function successful >>>>>>>> [compiler-2:08780] orte_rml_base_select: initializing rml component oob >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 30 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 15 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 32 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 33 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 5 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 10 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 12 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 9 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 34 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 2 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 21 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 22 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 45 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 46 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 1 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> [compiler-2:08780] [[42202,0],0] posting recv >>>>>>>> [compiler-2:08780] [[42202,0],0] posting persistent recv on tag 27 for >>>>>>>> peer [[WILDCARD],WILDCARD] >>>>>>>> Daemon was launched on node1-130-08 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-03 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-05 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-02 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-01 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-04 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-07 - beginning to initialize >>>>>>>> Daemon was launched on node1-130-06 - beginning to initialize >>>>>>>> Daemon [[42202,0],3] checking in as pid 7178 on host node1-130-03 >>>>>>>> [node1-130-03:07178] [[42202,0],3] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],2] checking in as pid 13581 on host node1-130-02 >>>>>>>> [node1-130-02:13581] [[42202,0],2] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],1] checking in as pid 17220 on host node1-130-01 >>>>>>>> [node1-130-01:17220] [[42202,0],1] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],5] checking in as pid 6663 on host node1-130-05 >>>>>>>> [node1-130-05:06663] [[42202,0],5] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],8] checking in as pid 6683 on host node1-130-08 >>>>>>>> [node1-130-08:06683] [[42202,0],8] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],7] checking in as pid 7877 on host node1-130-07 >>>>>>>> [node1-130-07:07877] [[42202,0],7] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],4] checking in as pid 7735 on host node1-130-04 >>>>>>>> [node1-130-04:07735] [[42202,0],4] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> Daemon [[42202,0],6] checking in as pid 8451 on host node1-130-06 >>>>>>>> [node1-130-06:08451] [[42202,0],6] orted: up and running - waiting for >>>>>>>> commands! >>>>>>>> srun: error: node1-130-03: task 2: Exited with exit code 1 >>>>>>>> srun: Terminating job step 657040.1 >>>>>>>> srun: error: node1-130-02: task 1: Exited with exit code 1 >>>>>>>> slurmd[node1-130-04]: *** STEP 657040.1 KILLED AT 2014-08-12T12:59:07 >>>>>>>> WITH SIGNAL 9 *** >>>>>>>> slurmd[node1-130-07]: *** STEP 657040.1 KILLED AT 2014-08-12T12:59:07 >>>>>>>> WITH SIGNAL 9 *** >>>>>>>> slurmd[node1-130-06]: *** STEP 657040.1 KILLED AT 2014-08-12T12:59:07 >>>>>>>> WITH SIGNAL 9 *** >>>>>>>> srun: Job step aborted: Waiting up to 2 seconds for job step to finish. >>>>>>>> srun: error: node1-130-01: task 0: Exited with exit code 1 >>>>>>>> srun: error: node1-130-05: task 4: Exited with exit code 1 >>>>>>>> srun: error: node1-130-08: task 7: Exited with exit code 1 >>>>>>>> srun: error: node1-130-07: task 6: Exited with exit code 1 >>>>>>>> srun: error: node1-130-04: task 3: Killed >>>>>>>> srun: error: node1-130-06: task 5: Killed >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> An ORTE daemon has unexpectedly failed after launch and before >>>>>>>> communicating back to mpirun. This could be caused by a number >>>>>>>> of factors, including an inability to create a connection back >>>>>>>> to mpirun due to a lack of common network interfaces and/or no >>>>>>>> route found between them. Please check network connectivity >>>>>>>> (including firewalls and network routing requirements). >>>>>>>> -------------------------------------------------------------------------- >>>>>>>> [compiler-2:08780] [[42202,0],0] orted_cmd: received halt_vm cmd >>>>>>>> [compiler-2:08780] mca: base: close: component oob closed >>>>>>>> [compiler-2:08780] mca: base: close: unloading component oob >>>>>>>> [compiler-2:08780] [[42202,0],0] TCP SHUTDOWN >>>>>>>> [compiler-2:08780] mca: base: close: component tcp closed >>>>>>>> [compiler-2:08780] mca: base: close: unloading component tcp >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/users/2014/08/24987.php >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> Link to this post: >>>>>>>> http://www.open-mpi.org/community/lists/users/2014/08/24988.php >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Squyres >>>>>>> jsquy...@cisco.com >>>>>>> For corporate legal information go to: >>>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Squyres >>>>>> jsquy...@cisco.com >>>>>> For corporate legal information go to: >>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2014/08/25001.php >>>>> >>>>> >>>>>-- >>>>>Jeff Squyres >>>>>jsquy...@cisco.com >>>>>For corporate legal information go to: >>>>>http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> >>>> >>>> >>>> >>>>_______________________________________________ >>>>users mailing list >>>>us...@open-mpi.org >>>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>Link to this post: >>>>http://www.open-mpi.org/community/lists/users/2014/08/25086.php >>> >>>_______________________________________________ >>>users mailing list >>>us...@open-mpi.org >>>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>Link to this post: >>>http://www.open-mpi.org/community/lists/users/2014/08/25093.php >> >>_______________________________________________ >>users mailing list >>us...@open-mpi.org >>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>Link to this post: >>http://www.open-mpi.org/community/lists/users/2014/08/25094.php >_______________________________________________ >users mailing list >us...@open-mpi.org >Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >Link to this post: >http://www.open-mpi.org/community/lists/users/2014/08/25095.php
