Re: [OMPI users] --mca btl_openib_if_include
On Oct 16, 2008, at 9:10 PM, Mostyn Lewis wrote: OpenMPI says for a: mpirun --prefix /tools/openmpi/1.4a1r19757_svn/connectx/gcc64/4.1.2/ openib/rh_EL_4/x86_64/xeon -x LD_LIBRARY_PATH --mca btl_openib_verbose 1 --mca btl openib,self --mca btl_openib_if_include "mlx4_0:1,mlx4_1:1" -np 4 -machinefile dhosts ./IMB-MPI1.openmpi -- WARNING: One or more nonexistent OpenFabrics devices/ports were specified: Host: r4450_3 MCA parameter:mca_btl_if_include Nonexistent entities: "mlx4_0:1,mlx4_1:1" I'm unable to replicate this problem. There might be some kind of bug in the if_include parsing code, I guess, but I can't make it happen on my machines. Can you dig into this code a bit? The code in question is in ompi/mca/btl/openib/ btl_openib_component.c:get_port_list(). The general scheme of that routine is as follows: - mca_btl_openib_component.if_list is an argv-style array of the items listed in btl_openib_if_include. - we call get_port_list() for each device that is found - we compare each item in .if_list to the device name and device_name:port combination to see if it matches - if we match, we include/exclude the device or port - we then remove the entry from the .if_list Later, if there are any entries left in .if_list (_component.c:2257), then we didn't find them and issue the warning. Can dig into why items are being left on the .if_list? One thing I will mention; it looks like the help message may be a little ambiguous -- the ports aren't necessarily nonexistent, they could also be non-ACTIVE. From your ibstatus output, it doesn't look like this is the case, though (I assume the ibstatus output you showed was from the r4450_3 host, right?). I'll go update that help message to be a bit more clear. FWIW, OMPI should normally silently ignore the DOWN ports and just run over the ACTIVE ports if you don't specify an _if_include list. But regardless, it would be good to solve this issue -- it's a bit troubling that you appear to be specifying ACTIVE ports and OMPI still issues a warning. -- Jeff Squyres Cisco Systems
Re: [OMPI users] OPAL_PREFIX is not passed to remote node in pls_rsh_module.c
On Oct 17, 2008, at 6:00 AM, Teng Lin wrote: It seems to work after I patched pls_rsh_module.c --- pls_rsh_module.c.orig 2008-10-16 17:15:32.0 -0400 +++ pls_rsh_module.c2008-10-16 17:15:51.0 -0400 @@ -989,7 +989,7 @@ "%s/%s/%s", (opal_prefix != NULL ? "OPAL_PREFIX=" : ""), (opal_prefix != NULL ? opal_prefix : ""), - (opal_prefix != NULL ? " ;" : ""), + (opal_prefix != NULL ? " ; export OPAL_PREFIX ; " : ""), prefix_dir, bin_base, prefix_dir, lib_base, prefix_dir, bin_base, Another workaround is to add export OPAL_PREFIX into $HOME/.bashrc. Jeff, is this a bug in the code? Or there is a reason that OPAL_PREFIX is not exported for sh/bash? Yes, this is a bug -- I think your patch is correct. Sorry about that, but thanks for the patch! -- Jeff Squyres Cisco Systems
Re: [OMPI users] OpenMPI portability problems: debug info isn't helpful
On Oct 16, 2008, at 1:39 PM, Aleksej Saushev wrote: [asau.local:04648] opal_ifinit: ioctl(SIOCGIFFLAGS) failed with errno=6 [snip] Why don't you use strerror(3) to print errno value explanation? I don't know; whoever wrote that just chose not to there. Shrug. From : #define ENXIO 6 /* Device not configured */ It seems that I have to debug network interface probing, how should I use *_output subroutines so that they do print? opal_output(0, "...printf-like stuff here...", ...printf varargs...); Should always output. Stream 0 is our "unconditional" stream; it goes to stderr. You'll see opal_output(0) output from mpirun and MPI apps; we usually close the stderr from orted's by default. Make sure that you re-build and re-install OMPI properly or your output won't show up. You can either do a top-level "make install" (which might be lengthy, depending on the speed of your system(s), and especially if you're iteratively editing opal/util/ if.c), or you can do the following: cd top-ompi-src-dir cd opal make make install-am If you built OMPI with shared libraries, that should be sufficient to recompile opal/util/if.c, rebuild libopen-pal.so, and re-install it. Or, if you *really* want to streamline, you can do something like this: cd top-ompi-src-dir cd opal cd util && make && cd .. && make install-am And then repeat that last step as necessary (this is the kind of thing I do when I am editing a single .c file in one of OMPI's libs). -- Jeff Squyres Cisco Systems
Re: [OMPI users] MPI_ERR_TRUNCATE
On Oct 17, 2008, at 6:03 PM, Nick Collier wrote: And under some conditions, I get the error: [3] [belafonte.home:04938] *** An error occurred in MPI_Wait [3] [belafonte.home:04938] *** on communicator MPI_COMM_WORLD [3] [belafonte.home:04938] *** MPI_ERR_TRUNCATE: message truncated [3] [belafonte.home:04938] *** MPI_ERRORS_ARE_FATAL (goodbye) When I do get the error, tracking the send and receive counts shows them as equal. And what I don't understand is that the receive_complete function in the above executes and the recv Struct actually contains the data that was sent. So, I'm confused about the error and what its trying to tell me as it looks like everything worked OK. Perhaps it's a race condition? Remember that MPI_Wait triggers OMPI's general progression engine -- so it may not be *this* receive that is the problem; it could be some other pending receive that has a mismatched send/receive length. -- Jeff Squyres Cisco Systems
Re: [OMPI users] --mca btl_openib_if_include
Jeff, I traced this and it was the quote marks in "mlx4_0:1,mlx4_1:1" - they were passed in and caused the mismatch :-( Sorry about that. Regards, DM On Sat, 18 Oct 2008, Jeff Squyres wrote: On Oct 16, 2008, at 9:10 PM, Mostyn Lewis wrote: OpenMPI says for a: mpirun --prefix /tools/openmpi/1.4a1r19757_svn/connectx/gcc64/4.1.2/openib/rh_EL_4/x86_64/xeon -x LD_LIBRARY_PATH --mca btl_openib_verbose 1 --mca btl openib,self --mca btl_openib_if_include "mlx4_0:1,mlx4_1:1" -np 4 -machinefile dhosts ./IMB-MPI1.openmpi -- WARNING: One or more nonexistent OpenFabrics devices/ports were specified: Host: r4450_3 MCA parameter:mca_btl_if_include Nonexistent entities: "mlx4_0:1,mlx4_1:1" I'm unable to replicate this problem. There might be some kind of bug in the if_include parsing code, I guess, but I can't make it happen on my machines. Can you dig into this code a bit? The code in question is in ompi/mca/btl/openib/btl_openib_component.c:get_port_list(). The general scheme of that routine is as follows: - mca_btl_openib_component.if_list is an argv-style array of the items listed in btl_openib_if_include. - we call get_port_list() for each device that is found - we compare each item in .if_list to the device name and device_name:port combination to see if it matches - if we match, we include/exclude the device or port - we then remove the entry from the .if_list Later, if there are any entries left in .if_list (_component.c:2257), then we didn't find them and issue the warning. Can dig into why items are being left on the .if_list? One thing I will mention; it looks like the help message may be a little ambiguous -- the ports aren't necessarily nonexistent, they could also be non-ACTIVE. From your ibstatus output, it doesn't look like this is the case, though (I assume the ibstatus output you showed was from the r4450_3 host, right?). I'll go update that help message to be a bit more clear. FWIW, OMPI should normally silently ignore the DOWN ports and just run over the ACTIVE ports if you don't specify an _if_include list. But regardless, it would be good to solve this issue -- it's a bit troubling that you appear to be specifying ACTIVE ports and OMPI still issues a warning. -- Jeff Squyres Cisco Systems ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
[OMPI users] Can 2 IB HCAs give twice the bandwidth?
Can OpenMPI do like Scali and MVAPICH2 and utilize 2 IB HCAs per machine to approach double the bandwidth on simple tests such as IMB PingPong? Regards, DM
