[OMPI users] OMPI-1.3.2, openib/iWARP(cxgb3) problem: PML add procs failed (Unreachable)
I am trying to run NetPIPE-3.7.1 NPmpi using Open MPI version 1.3.2 with the openib btl in an OFED-1.4 environment. The system environment is two Linux (2.6.27) ppc64 blades, each with one Chelsio RNIC device, interconnected by a 10GbE switch. The problem is that I cannot (using Open MPI) establish connections between the two MPI ranks. I have already read the OMPI FAQ entries and searched for similar problems reported to this email list without success. I do have a compressed config.log that I can provide separately (it is 80KB in size so I'll spare everyone here). I also have the output of ompi_info --all that I can share. I can successfully run small diagnostic programs such as rping, ib_rdma_bw, ib_rdma_lat, etc. between the same two blades. I can also run NPmpi using another MPI library (MVAPICH2) and the Chelsio/iWARP interface. Here is the one example mpirun command line I used: mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > outfile1 2>&1 and its output: -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: aae1 Local device: cxgb3_0 CPCs attempted: oob, xoob, rdmacm -- -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: aae4 Local device: cxgb3_0 CPCs attempted: oob, xoob, rdmacm -- -- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3115,1],0]) is on host: aae4 Process 2 ([[3115,1],1]) is on host: aae1 BTLs attempted: self Your MPI job is now going to abort; sorry. -- -- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3115,1],1]) is on host: aae1 Process 2 ([[3115,1],0]) is on host: aae4 BTLs attempted: self Your MPI job is now going to abort; sorry. -- -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) -- *** An error occurred in MPI_Init -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) -- *** An error occurred in MPI_Init *** before MPI was initialized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) *** before MPI was initialized [aae1:6598] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed! *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [aae4:19434] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed!
Re: [OMPI users] OMPI-1.3.2, openib/iWARP(cxgb3) problem: PML add procs failed (Unreachable)
Thanks Jon. I have some responses inline. Jon Mason wrote: On Wed, May 06, 2009 at 12:15:19PM -0400, Ken Cain wrote: I am trying to run NetPIPE-3.7.1 NPmpi using Open MPI version 1.3.2 with the openib btl in an OFED-1.4 environment. The system environment is two Linux (2.6.27) ppc64 blades, each with one Chelsio RNIC device, interconnected by a 10GbE switch. The problem is that I cannot (using Open MPI) establish connections between the two MPI ranks. I have already read the OMPI FAQ entries and searched for similar problems reported to this email list without success. I do have a compressed config.log that I can provide separately (it is 80KB in size so I'll spare everyone here). I also have the output of ompi_info --all that I can share. I can successfully run small diagnostic programs such as rping, ib_rdma_bw, ib_rdma_lat, etc. between the same two blades. I can also run NPmpi using another MPI library (MVAPICH2) and the Chelsio/iWARP interface. Here is the one example mpirun command line I used: mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > outfile1 2>&1 and its output: -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: aae1 Local device: cxgb3_0 CPCs attempted: oob, xoob, rdmacm -- -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: aae4 Local device: cxgb3_0 CPCs attempted: oob, xoob, rdmacm -- -- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3115,1],0]) is on host: aae4 Process 2 ([[3115,1],1]) is on host: aae1 BTLs attempted: self Your MPI job is now going to abort; sorry. -- -- At least one pair of MPI processes are unable to reach each other for MPI communications. This means that no Open MPI device has indicated that it can be used to communicate between these processes. This is an error; Open MPI requires that all MPI processes be able to reach each other. This error can sometimes be the result of forgetting to specify the "self" BTL. Process 1 ([[3115,1],1]) is on host: aae1 Process 2 ([[3115,1],0]) is on host: aae4 BTLs attempted: self Your MPI job is now going to abort; sorry. -- -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) -- *** An error occurred in MPI_Init -- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) -- *** An error occurred in MPI_Init *** before MPI was initialized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) *** before MPI was initialized [aae1:6598] Abort before MPI_INIT completed successfully; not able to guarantee that all other proce
Re: [OMPI users] OMPI-1.3.2, openib/iWARP(cxgb3) problem: PML add procs failed (Unreachable)
Jon Mason wrote: On Wed, May 06, 2009 at 01:20:48PM -0400, Ken Cain wrote: Thanks Jon. I have some responses inline. Jon Mason wrote: On Wed, May 06, 2009 at 12:15:19PM -0400, Ken Cain wrote: I am trying to run NetPIPE-3.7.1 NPmpi using Open MPI version 1.3.2 with the openib btl in an OFED-1.4 environment. The system environment is two Linux (2.6.27) ppc64 blades, each with one Chelsio RNIC device, interconnected by a 10GbE switch. The problem is that I cannot (using Open MPI) establish connections between the two MPI ranks. I have already read the OMPI FAQ entries and searched for similar problems reported to this email list without success. I do have a compressed config.log that I can provide separately (it is 80KB in size so I'll spare everyone here). I also have the output of ompi_info --all that I can share. I can successfully run small diagnostic programs such as rping, ib_rdma_bw, ib_rdma_lat, etc. between the same two blades. I can also run NPmpi using another MPI library (MVAPICH2) and the Chelsio/iWARP interface. Here is the one example mpirun command line I used: mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > outfile1 2>&1 and its output: -- No OpenFabrics connection schemes reported that they were able to be used on a specific port. As such, the openib BTL (OpenFabrics support) will be disabled for this port. Local host: aae1 Local device: cxgb3_0 CPCs attempted: oob, xoob, rdmacm -- [snip] Process 1 ([[3115,1],0]) is on host: aae4 Process 2 ([[3115,1],1]) is on host: aae1 BTLs attempted: self Your MPI job is now going to abort; sorry. -- [snip] PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) -- *** An error occurred in MPI_Init -- [snip] *** An error occurred in MPI_Init *** before MPI was initialized *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) *** before MPI was initialized [aae1:6598] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed! *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) [aae4:19434] Abort before MPI_INIT completed successfully; not able to guarantee that all other processes were killed! -- mpirun has exited due to process rank 0 with PID 19434 on node aae4 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -- Here is the another mpirun command I used (adding verbosity and more specific btl parameters): mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self,sm --mca btl_base_verbose 10 --mca btl_openib_verbose 10 --mca btl_openib_if_include cxgb3_0:1 --mca btl_openib_cpc_include rdmacm --mca btl_openib_device_type iwarp --mca btl_openib_max_btls 1 --mca mpi_leave_pinned 1 --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > ~/outfile2 2>&1 It looks like you are only using 1 port on the Chelsio RNIC. Based on the messages above, It looks like it might be the wrong port. Is there a reason why you are excluding it? Also, you might try the TCP btl and verify that it works correctly in the testcase (as a point of reference). Thanks, Jon Yes we only have one port connected. The cxgb3 device is associated with eth2 and eth3. Only eth2 is configured with a static IP address. To be sure I didn't choose the wrong OFED device I tried the same command line, changing btl_openib_if_include to cxgb3_0:2 (instead of my original choice cxgb3_0:1). I got the same result in this experiment. The same result occurs when I ask only for cxgb3_0 (no particular port). To test with the TCP btl I changed both of my mpirun commands so that I added tcp to the --mca btl list (keeping openib and the others). In both cases the NPmpi application runs to completion (using a TCP/IP transport not iWARP). In the first (simpler) mpirun command I get the same "No OpenFabrics connection schemes ..." warning message (followed by successful run to completion as noted). In the second mpirun command I get no particular warning messages and run to completion. Hmm...If you are just adding tcp and keeping openib in the btl grouping, it might be possible that it is using the openib btl with the simplier commandline. C
Re: [OMPI users] OMPI-1.3.2, openib/iWARP(cxgb3) problem: PML add procs failed (Unreachable)
Jeff Squyres wrote: On May 6, 2009, at 4:45 PM, Ken Cain wrote: Is it possible for OMPI to generate output at runtime indicating exactly what btl(s) will be used? At present, we only have a fairly lame system to do this. We wanted to print out a connection map in v1.3, but it didn't happen -- this feature has been re-targeted for v1.5: https://svn.open-mpi.org/trac/ompi/ticket/1207 It's unfortunately a surprisingly complex issue; one reason that it's "hard" is that OMPI lazily makes connections and supports striping across multiple networks. Hence, to make a completely accurate map, OMPI has to guarantee to make *all* network connections and then gather all the connection information back to MPI_COMM_WORLD rank 0 to print out. What OMPI does today is that if you specifically ask for a high-speed network and we're unable to find one, we'll warn about it (because if you asked for it, you likely really want to use it -- if there isn't one, that's likely a problem). So if you: mpirun --mca btl openib,sm,self,tcp ... And OMPI doesn't find any active OpenFabrics ports, it'll print a warning. Removing tcp below brings me back to the original simple command line that fails with the output shown above (indicating that openib btl will be disabled): mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 It looks like you're having two problems: 1. The RDMACM connector in OMPI decides that it can't be used: mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > outfile1 2>&1 > -- > No OpenFabrics connection schemes reported that they were able to be > used on a specific port. As such, the openib BTL (OpenFabrics > support) will be disabled for this port. > > Local host: aae1 > Local device: cxgb3_0 > CPCs attempted: oob, xoob, rdmacm > -- *** Can you re-run this scenario with --mca btl_base_verbose 50? I'd like to see why the RDMA CM CPC disqualified itself. Jeff, thank you very much for taking a look at this. I have re-run with increased verbosity in three different scenarios: 1) simple command line with verbosity mpirun --mca orte_base_help_aggregate 0 --mca btl_base_verbose 50 --mca btl_openib_verbose 50 --mca btl openib,self --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > ~/outfile3 2>&1 interesting output below indicates rdmacm IP address not found on port (showing output of one rank below, but we get the same output by the other MPI rank as well): [aae4:30924] openib BTL: oob CPC only supported on InfiniBand; skipped on device cxgb3_0 [aae4:30924] openib BTL: xoob CPC only supported with XRC receive queues; skipped on device cxgb3_0 [aae4:30924] openib BTL: rdmacm CPC available for use on cxgb3_0 [aae4:30924] openib BTL: oob CPC only supported on InfiniBand; skipped on device cxgb3_0 [aae4:30924] openib BTL: xoob CPC only supported with XRC receive queues; skipped on device cxgb3_0 [aae4:30924] openib BTL: rdmacm IP address not found on port [aae4:30924] openib BTL: rdmacm CPC unavailable for use on cxgb3_0; skipped [aae4:30924] select: init of component openib returned failure [aae4:30924] select: module openib unloaded 2) more complex command line requesting to use cxgb3_0:1 (the one I believe is physically connected + configured with an IP address): mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --mca btl_base_verbose 50 --mca btl_openib_verbose 50 --mca btl_openib_if_include cxgb3_0:1 --mca btl_openib_cpc_include rdmacm --mca btl_openib_device_type iwarp --mca btl_openib_max_btls 1 --mca mpi_leave_pinned 1 --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > ~/outfile4_cxgb3_0_port1 2>&1 output (one rank shown, both print the same pattern): [aae4:30929] select: initializing btl component openib [aae4:30929] openib BTL: rdmacm CPC available for use on cxgb3_0 [aae4:30929] select: init of component openib returned success but then! PML add procs failed --> Returned "Unreachable" (-12) instead of "Success" (0) 3) more complex command line requesting to use cxgb3_0:2 (the one I believe is not physically connected and not configured with an IP address): mpirun --mca orte_base_help_aggregate 0 --mca btl openib,self --mca btl_base_verbose 50 --mca btl_openib_verbose 50 --mca btl_openib_if_include cxgb3_0:2 --mca btl_openib_cpc_include rdmacm --mca btl_openib_device_type iwarp --mca btl_openib_max_btls 1 --mca mpi_leave_pinned 1 --hostfile ~/1usrv_ompi_machfile -np 2 ./NPmpi -p0 -l 1 -u 1024 > ~/outfi
