Re: [OMPI users] error messages for btl components that aren't loaded
This is due to the way the OMPI finds and loads modules. What actually happens is that OMPI looks for *all* modules of a given type and dlopen's them. It then applies the filter of which components are desired and dlclose's all the undesired ones. It certainly would be better to apply the filter earlier and only open the desired modules. We actually identified this behavior quite a while ago, but never put a high priority on fixing it because we didn't think it would be much of an issue (because most people build/run in homogeneous environments). But pending resource availability, I agree that this behavior is sub-optimal and should be fixed. I'll enter this issue on the bug tracker so that we don't forget about it. Can you -- and anyone else in similar circumstances -- let me know how common this scenario is? There is one workaround, however. The MCA parameter mca_component_show_load_errors defaults to a "1" value. When it's 1, all warnings regarding the loading of components are displayed (i.e., the messages you're seeing). Setting this value to 0 will disable the messages. However, you won't see *any* messages about components not loading. For example, if you have components that you think should be loading but are not, you won't be notified. That being said, these messages are not usually a concern for end-users, however -- they are typically more useful for the OMPI developers. For example, if a developer accidentally does something to make a plugin un-loadable (e.g., leaves a symbol out), having these messages displayed at mpirun time can be *very* useful. Plugins that are shipped in a tarball hopefully do not suffer from such issues :-), and usually have rpath information compiled in them so even LD_LIBRARY_PATH issues shouldn't be much of a problem. > -Original Message- > From: users-boun...@open-mpi.org > [mailto:users-boun...@open-mpi.org] On Behalf Of Patrick Jessee > Sent: Wednesday, June 28, 2006 5:18 PM > To: Open MPI Users > Subject: [OMPI users] error messages for btl components that > aren't loaded > > > Hello. I'm getting some odd error messages in certain situations > associated with the btl components (happens with both 1.0.2 > and 1.1). > When certain btl components are NOT loaded, openMPI issues error > messages associated with those very components. For > instance, consider > an application that is built with an openMPI installation that was > configured with mvapi and mx (in addition to tcp,sm,self). If that > application is taken to a system that does not have mvapi and mx > interconnects installed and is explicitly started for TCP by using > "--mca btl self,tcp,sm", then the following comes from openMPI: > > [devi01:01659] mca: base: component_find: unable to open: libvapi.so: > cannot open shared object file: No such file or directory (ignored) > [devi01:01659] mca: base: component_find: unable to open: libvapi.so: > cannot open shared object file: No such file or directory (ignored) > [devi01:01659] mca: base: component_find: unable to open: > libmyriexpress.so: cannot open shared object file: No such file or > directory (ignored) > [devi02:31845] mca: base: component_find: unable to open: libvapi.so: > cannot open shared object file: No such file or directory (ignored) > [devi02:31845] mca: base: component_find: unable to open: libvapi.so: > cannot open shared object file: No such file or directory (ignored) > [devi02:31845] mca: base: component_find: unable to open: > libmyriexpress.so: cannot open shared object file: No such file or > directory (ignored) > > These are not fatal, but they definitely give the wrong > impression that > something is not right. The "--mca btl self,tcp,sm" option > should tell > openMPI only to load loopback, tcp, and shared memory components (as > these are the only btl components that should be operational on the > system). The mvapi and mx components (which need libvapi.so and > libmyriexpress.so, respectively), should not be loaded and thus > libvapi.so and libmyriexpress.so should not be needed or even > searched > for. The same thing happens with "--mca btl ^mvapi,mx". > Interestingly, > even on a system that does have MX, the libmyriexpress.so > errors show up > if the mx btl component is not loaded. > > Does anyone know (a) why openMPI is complaining about a > shared library > from a component that isn't even loaded, and (b) how to avoid the > seemingly superfluous error messages? Any help is greatly > appreciated. > > -Patrick > > >
Re: [OMPI users] keyval parser error after v1.1 upgrade
Ok, good -- that's what I was assuming had happened.
As Brian said, the stdin issue is not currently on our roadmap to fix in
the immediate future. But I added it to our bug tracker as an
un-milestoned issue so that we don't forget about it
https://svn.open-mpi.org/trac/ompi/ticket/167
Thanks for the report!
> -Original Message-
> From: users-boun...@open-mpi.org
> [mailto:users-boun...@open-mpi.org] On Behalf Of Patrick Jessee
> Sent: Thursday, June 29, 2006 5:12 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] keyval parser error after v1.1 upgrade
>
> Jeff,
>
> Sorry for the confusion. It's for the the
> "mca_oob_tcp_accept" thread.
> I mistakenly replied to the wrong message ("keyval parser").
>
> As for the "mca_oob_tcp_accept" thread, I have since found
> since found
> some more information on the problem (1.1 no longer works if stdin is
> closed; 1.0.2 was fine) and have reported this in another message
> ("Openmpi 1.1: startup problem caused by file descriptor state").
>
> Let me know if you need anymore information. Thanks for following up.
>
> -Patrick
>
>
> Jeff Squyres (jsquyres) wrote:
>
> >Patrick --
> >
> >I'm a little confused about your response. Are you replying to the
> >"keyval parser" thread (i.e., saying that you had the same problem as
> >Benjamin Landsteiner), or are you replying to the
> "mca_oob_tcp_accept"
> >thread?
> >
> >
> >
> >
> >>-Original Message-
> >>From: users-boun...@open-mpi.org
> >>[mailto:users-boun...@open-mpi.org] On Behalf Of Patrick Jessee
> >>Sent: Tuesday, June 27, 2006 11:57 AM
> >>To: Open MPI Users
> >>Subject: Re: [OMPI users] keyval parser error after v1.1 upgrade
> >>
> >>Michael Kluskens wrote:
> >>
> >>
> >>
> >>>If you have both OpenMPI 1.0.2 and 1.1 installed in
> separate areas
> >>>there are a lot of different ways to mess that up.
> >>>
> >>>Offhand:
> >>>
> >>>Did you configure with --prefix pointed at each of the
> >>>
> >>>
> >>different areas.
> >>
> >>
> >>>
> >>>
> >>>
> >>>
> >>Yes, --prefix was unique for each different area.
> >>
> >>
> >>
> >>>If both areas are in PATH or LD_LIBRARY_PATH or whatever
> >>>
> >>>
> >>environment
> >>
> >>
> >>>variable your compiler uses than things could be
> interesting (Intel
> >>>compiler using FPATH for include files).
> >>>
> >>>
> >>>
> >>>
> >>>
> >>Neither area was in the PATH, etc.
> >>
> >>
> >>
> >>>Michael
> >>>
> >>>On Jun 26, 2006, at 6:28 PM, Patrick Jessee wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> Michael,
>
> Hello. Thanks for the response. We do clean configures
> and makes
> under /tmp, and install in a completely separate area so I don't
> see how anything from 1.0.2 could be left over in the 1.1
> installation. We aren't installing 1.1 over 1.0.2. 1.1 is
> configured, built, and installed in a completely different area.
>
> -Patrick
>
> Michael Kluskens wrote:
>
>
>
>
>
> >You may have to properly uninstall OpenMPI 1.0.2 before
> >installing OpenMPI 1.1
> >
> >This was an issue in the past.
> >
> >I would recommend you go into your OpenMPI 1.1 directory
> >
> >
> >>and type
> >>
> >>
> >"make uninstall", then if you have it go into your
> OpenMPI 1.0.2
> >directory and do the same. If you don't have a directory with
> >OpenMPI 1.0.2 configured already then either rebuild
> >
> >
> >>OpenMPI 1.0.2
> >>
> >>
> >or go into /usr/local and identify all remaining OpenMPI
> >directories and components and remove them. Basically
> >
> >
> >>you should
> >>
> >>
> >find directories modified when you installed OpenMPI 1.1
> >
> >
> >>(or when
> >>
> >>
> >you uninstalled it) and you may find components dated
> from when
> >you installed OpenMPI 1.0.2.
> >
> >Michael
> >
> >On Jun 26, 2006, at 4:34 PM, Benjamin Landsteiner wrote:
> >
> >
> >
> >
> >
> >
> >>Hello all,
> >>I recently upgraded to v1.1 of Open MPI and ran
> into a problem
> >>on my
> >>head node that I can't seem to solve. Upon running
> >>
> >>
> >>mpicc, mpiCC,
> >>
> >>
> >>mpic
> >>++, and so forth, I get an error like this:
> >>
> >>
> >>
> >>
> >>
> >___
> >users mailing list
> >us...@open-mpi.org
> >http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> >
> >
> >
> >
> >
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
>
> >>>___
Re: [OMPI users] OpenMpi 1.1 and Torque 2.1.1
There was a bug in early Torque 2.1.x versions (I'm afraid I don't remember which one) that -- I think -- had something to do with a faulty poll() implementation. Whatever the problem was, it caused all TM launchers to fail on OSX. Can you see if the Torque-included tool pbsdsh works properly? It uses the same API that Open MPI does (the "tm" api). If pbsdsh fails, I suspect you're looking at a Torque bug. I know that Garrick S. has since fixed the problem in the Torque code base; I don't know if they've had a release since then that included the fix. If pbsdsh works, let us know and we'll track this down further. From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Justin Bronder Sent: Thursday, June 29, 2006 5:19 PM To: us...@open-mpi.org Subject: [OMPI users] OpenMpi 1.1 and Torque 2.1.1 I'm having trouble getting OpenMPI to execute jobs when submitting through Torque. Everything works fine if I am to use the included mpirun scripts, but this is obviously not a good solution for the general users on the cluster. I'm running under OS X 10.4, Darwin 8.6.0. I configured OpenMpi with: export CC=/opt/ibmcmp/vac/6.0/bin/xlc export CXX=/opt/ibmcmp/vacpp/6.0/bin/xlc++ export FC=/opt/ibmcmp/xlf/8.1/bin/xlf90_r export F77=/opt/ibmcmp/xlf/8.1/bin/xlf_r export LDFLAGS=-lSystemStubs export LIBTOOL=glibtool PREFIX=/usr/local/ompi-xl ./configure \ --prefix=$PREFIX \ --with-tm=/usr/local/pbs/ \ --with-gm=/opt/gm \ --enable-static \ --disable-cxx I also had to employ the fix listed in: http://www.open-mpi.org/community/lists/users/2006/04/1007.php I've attached the output of ompi_info while in an interactive job. Looking through the list, I can at least save a bit of trouble by listing what does work. Anything outside of Torque seems fine. From within an interactive job, pbsdsh works fine, hence the earlier problems with poll are fixed. Here is the error that is reported when I attemt to run hostname on one processor: node96:/usr/src/openmpi-1.1 jbronder$ /usr/local/ompi-xl/bin/mpirun -np 1 -mca pls_tm_debug 1 /bin/hostname [node96.meldrew.clusters.umaine.edu:00850] pls:tm: final top-level argv: [node96.meldrew.clusters.umaine.edu:00850] pls:tm: orted --no-daemonize --bootproxy 1 --name --num_procs 2 --vpid_start 0 --nodename --universe jbron...@node96.meldrew.clusters.umaine.edu:default-universe --nsreplica "0.0.0;tcp://10.0.1.96:49395" --gprreplica "0.0.0;tcp://10.0.1.96:49395" [node96.meldrew.clusters.umaine.edu:00850] pls:tm: Set prefix:/usr/local/ompi-xl [node96.meldrew.clusters.umaine.edu:00850] pls:tm: launching on node localhost [node96.meldrew.clusters.umaine.edu:00850] pls:tm: resetting PATH: /usr/local/ompi-xl/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/pbs/bin: /usr/local/mpiexec/bin:/opt/ibmcmp/xlf/8.1/bin:/opt/ibmcmp/vac/6.0/bin:/ opt/ibmcmp/vacpp/6.0/bin:/opt/gm/bin:/opt/fms/bin [node96.meldrew.clusters.umaine.edu:00850] pls:tm: found /usr/local/ompi-xl/bin/orted [node96.meldrew.clusters.umaine.edu:00850] pls:tm: not oversubscribed -- setting mpi_yield_when_idle to 0 [node96.meldrew.clusters.umaine.edu:00850] pls:tm: executing: orted --no-daemonize --bootproxy 1 --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename localhost --universe jbron...@node96.meldrew.clusters.umaine.edu:default-universe --nsreplica "0.0.0;tcp://10.0.1.96:49395" --gprreplica "0.0.0;tcp://10.0.1.96:49395" [node96.meldrew.clusters.umaine.edu:00850] pls:tm: start_procs returned error -13 [node96.meldrew.clusters.umaine.edu:00850] [0,0,0] ORTE_ERROR_LOG: Not found in file rmgr_urm.c at line 184 [node96.meldrew.clusters.umaine.edu:00850] [0,0,0] ORTE_ERROR_LOG: Not found in file rmgr_urm.c at line 432 [node96.meldrew.clusters.umaine.edu:00850] mpirun: spawn failed with errno=-13 node96:/usr/src/openmpi-1.1 jbronder$ My thanks for any help in advance, Justin Bronder.
Re: [OMPI users] mca_btl_tcp_frag_send: writev failed with errno=110
Jeff Thanks for the reply; I realize you guys must be really busy with the recent release of openmpi. I tried 1.1 and I don't get error messages any more. But the code now hangs; no error or exit. So I am not sure if this is the same issue or something else. I am enclosing my source code. I compiled with icc and linked against an icc compiled version of openmpi-1.1. My program is a set of network benchmarks (a crude kind of netpipe) that checks typical message passing patterns in my application codes. Typical output is: 32 CPU's: sync call time = 1003.0time rate (Mbytes/s) bandwidth (MBits/s) loop buffers size XC XE GS MS XC XE GS MS XC XE GS MS 1 6416384 2.48e-02 1.99e-02 1.21e+00 3.88e-02 4.23e+01 5.28e+01 8.65e-01 2.70e+01 1.08e+04 1.35e+04 4.43e+02 1.38e+04 2 6416384 2.17e-02 2.09e-02 1.21e+00 4.10e-02 4.82e+01 5.02e+01 8.65e-01 2.56e+01 1.23e+04 1.29e+04 4.43e+02 1.31e+04 3 6416384 2.20e-02 1.99e-02 1.01e+00 3.95e-02 4.77e+01 5.27e+01 1.04e+00 2.65e+01 1.22e+04 1.35e+04 5.33e+02 1.36e+04 4 6416384 2.16e-02 1.96e-02 1.25e+00 4.00e-02 4.85e+01 5.36e+01 8.37e-01 2.62e+01 1.24e+04 1.37e+04 4.28e+02 1.34e+04 5 6416384 2.25e-02 2.00e-02 1.25e+00 4.07e-02 4.66e+01 5.24e+01 8.39e-01 2.57e+01 1.19e+04 1.34e+04 4.30e+02 1.32e+04 6 6416384 2.19e-02 1.99e-02 1.29e+00 4.05e-02 4.79e+01 5.28e+01 8.14e-01 2.59e+01 1.23e+04 1.35e+04 4.17e+02 1.33e+04 7 6416384 2.19e-02 2.06e-02 1.25e+00 4.03e-02 4.79e+01 5.09e+01 8.38e-01 2.60e+01 1.23e+04 1.30e+04 4.29e+02 1.33e+04 8 6416384 2.24e-02 2.06e-02 1.25e+00 4.01e-02 4.69e+01 5.09e+01 8.39e-01 2.62e+01 1.20e+04 1.30e+04 4.30e+02 1.34e+04 9 6416384 4.29e-01 2.01e-02 6.35e-01 3.98e-02 2.45e+00 5.22e+01 1.65e+00 2.64e+01 6.26e+02 1.34e+04 8.46e+02 1.35e+04 10 6416384 2.16e-02 2.06e-02 8.87e-01 4.00e-02 4.85e+01 5.09e+01 1.18e+00 2.62e+01 1.24e+04 1.30e+04 6.05e+02 1.34e+04 Time is total for all 64 buffers. Rate is one way across one link (# of bytes/time). 1) XC is a bidirectional ring exchange. Each processor sends to the right and receives from the left 2) XE is an edge exchange. Pairs of nodes exchange data, with each one sending and receiving 3) GS is the MPI_AllReduce 4) MS is my version of MPI_AllReduce. It splits the vector into Np blocks (Np is # of processors); each processor then acts as a head node for one block. This uses the full bandwidth all the time, unlike AllReduce which thins out as it gets to the top of the binary tree. On a 64 node Infiniband system MS is about 5X faster than GS-in theory it would be 6X; ie log_2(64). Here it is 25X-not sure why so much. But MS seems to be the cause of the hangups with messages > 64K. I can run the other benchmarks OK,but this one seems to hang for large messages. I think the problem is at least partly due to the switch. All MS is doing is point to point communications, but unfortunately it sometimes requires a high bandwidth between ASIC's. It first it exchanges data between near neighbors in MPI_COMM_WORLD, but it must progressively span wider gaps between nodes as it goes up the various binary trees. After a while this requires extensive traffic between ASICS. This seems to be a problem on both my HP 2724 and the Extreme Networks Summit400t-48. I am currently working with Extreme to try to resolve the switch issue. As I say; the code ran great on Infiniband, but I think those switches have hardware flow control. Finally I checked the code again under LAM and it ran OK. Slow, but no hangs. To run the code compile and type: mpirun -np 32 -machinefile hosts src/netbench 8 The 8 means 2^8 bytes (ie 256K). This was enough to hang every time on my boxes. You can also edit the header file (header.h). MAX_LOOPS is how many times it runs each test (currently 10); NUM_BUF is the number of buffers in each test (must be more than number of processors), SYNC defines the global sync frequency-every SYNC buffers. NUM_SYNC is the number of sequential barrier calls it uses to determine the mean barrier call time. You can also switch the verious tests on and off, which can be useful for debugging Tony --- Tony Ladd Professor, Chemical Engineering University of Florida PO Box 116005 Gainesville, FL 32611-6005 Tel: 352-392-6509 FAX: 352-392-9513 Email: tl...@che.ufl.edu Web: http://ladd.che.ufl.edu src.tgz Description: application/compressed
Re: [OMPI users] OpenMpi 1.1 and Torque 2.1.1
Greetings, The bug with poll was fixed in the stable Torque 2.1.1 release, and I have checked again to make sure that pbsdsh does work. jbronder@meldrew-linux ~/src/hpl $ qsub -I -q default -l nodes=4:ppn=2 -l opsys=darwin qsub: waiting for job 312.ldap1.meldrew.clusters.umaine.edu to start qsub: job 312.ldap1.meldrew.clusters.umaine.edu ready node96:~ jbronder$ pbsdsh uname -a Darwin node96.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node96.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node94.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node94.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node95.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node95.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node93.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc Darwin node93.meldrew.clusters.umaine.edu 8.6.0 Darwin Kernel Version 8.6.0: Tue Mar 7 16:58:48 PST 2006; root:xnu-792.6.70.obj~1/RELEASE_PPC Power Macintosh powerpc node96:~ jbronder$ If there is anything else I should check, please let me know. Thanks, Justin Bronder. On 6/30/06, Jeff Squyres (jsquyres) wrote: There was a bug in early Torque 2.1.x versions (I'm afraid I don't remember which one) that -- I think -- had something to do with a faulty poll() implementation. Whatever the problem was, it caused all TM launchers to fail on OSX. Can you see if the Torque-included tool pbsdsh works properly? It uses the same API that Open MPI does (the "tm" api). If pbsdsh fails, I suspect you're looking at a Torque bug. I know that Garrick S. has since fixed the problem in the Torque code base; I don't know if they've had a release since then that included the fix. If pbsdsh works, let us know and we'll track this down further. -- *From:* users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] *On Behalf Of *Justin Bronder *Sent:* Thursday, June 29, 2006 5:19 PM *To:* us...@open-mpi.org *Subject:* [OMPI users] OpenMpi 1.1 and Torque 2.1.1 I'm having trouble getting OpenMPI to execute jobs when submitting through Torque. Everything works fine if I am to use the included mpirun scripts, but this is obviously not a good solution for the general users on the cluster. I'm running under OS X 10.4, Darwin 8.6.0. I configured OpenMpi with: export CC=/opt/ibmcmp/vac/6.0/bin/xlc export CXX=/opt/ibmcmp/vacpp/6.0/bin/xlc++ export FC=/opt/ibmcmp/xlf/8.1/bin/xlf90_r export F77=/opt/ibmcmp/xlf/8.1/bin/xlf_r export LDFLAGS=-lSystemStubs export LIBTOOL=glibtool PREFIX=/usr/local/ompi-xl ./configure \ --prefix=$PREFIX \ --with-tm=/usr/local/pbs/ \ --with-gm=/opt/gm \ --enable-static \ --disable-cxx I also had to employ the fix listed in: http://www.open-mpi.org/community/lists/users/2006/04/1007.php I've attached the output of ompi_info while in an interactive job. Looking through the list, I can at least save a bit of trouble by listing what does work. Anything outside of Torque seems fine. From within an interactive job, pbsdsh works fine, hence the earlier problems with poll are fixed. Here is the error that is reported when I attemt to run hostname on one processor: node96:/usr/src/openmpi-1.1 jbronder$ /usr/local/ompi-xl/bin/mpirun -np 1 -mca pls_tm_debug 1 /bin/hostname [node96.meldrew.clusters.umaine.edu:00850] pls:tm: final top-level argv: [node96.meldrew.clusters.umaine.edu:00850] pls:tm: orted --no-daemonize --bootproxy 1 --name --num_procs 2 --vpid_start 0 --nodename --universe jbron...@node96.meldrew.clusters.umaine.edu:default-universe --nsreplica "0.0.0;tcp://10.0.1.96:49395" --gprreplica "0.0.0 ;tcp://10.0.1.96:49395" [node96.meldrew.clusters.umaine.edu:00850] pls:tm: Set prefix:/usr/local/ompi-xl [node96.meldrew.clusters.umaine.edu:00850] pls:tm: launching on node localhost [node96.meldrew.clusters.umaine.edu:00850] pls:tm: resetting PATH: /usr/local/ompi-xl/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/pbs/bin:/usr/local/mpiexec/bin:/opt/ibmcmp/xlf/8.1/bin:/opt/ibmcmp/vac/6.0/bin:/opt/ibmcmp/vacpp/6.0/bin:/opt/gm/bin:/opt/fms/bin [node96.meldrew.clusters.umaine.edu:00850] pls:tm: found /usr/local/ompi-xl/bin/orted [node96.meldrew.clusters.umaine.edu:00850] pls:tm: not oversubscribed -- setting mpi_yiel
Re: [OMPI users] error messages for btl components that aren't loaded
Jeff, Thanks for the reply and your attention to this. Can you -- and anyone else in similar circumstances -- let me know how common this scenario is? I think this depends on the environment. For us and many other ISVs, it is very common. The build host is almost always physically different than the target systems, and the target systems usually only have a subset of the network hardware for which the application was originally configured (and may have drivers installed in different places). The application will be configured for all possible interconnects. On the individual target systems (each with possibly different interconnect types), the specific interconnect will be selected either by user input or by auto-detection. Thus, we would build for mx, gm, mvapi, openib, tcp, sm, ...; however, some systems may have only mx or only mvapi or neither. The MCA of OpenMPI seems like it is very well suited for such a development environment because certain components can be selectively activated at run-time depending on the system. Your idea of applying the filter earlier and only opening the desired modules sounds like an excellent approach. Thanks for considering the issue. Please let me know if I can provide any more information. -Patrick Jeff Squyres (jsquyres) wrote: This is due to the way the OMPI finds and loads modules. What actually happens is that OMPI looks for *all* modules of a given type and dlopen's them. It then applies the filter of which components are desired and dlclose's all the undesired ones. It certainly would be better to apply the filter earlier and only open the desired modules. We actually identified this behavior quite a while ago, but never put a high priority on fixing it because we didn't think it would be much of an issue (because most people build/run in homogeneous environments). But pending resource availability, I agree that this behavior is sub-optimal and should be fixed. I'll enter this issue on the bug tracker so that we don't forget about it. Can you -- and anyone else in similar circumstances -- let me know how common this scenario is? There is one workaround, however. The MCA parameter mca_component_show_load_errors defaults to a "1" value. When it's 1, all warnings regarding the loading of components are displayed (i.e., the messages you're seeing). Setting this value to 0 will disable the messages. However, you won't see *any* messages about components not loading. For example, if you have components that you think should be loading but are not, you won't be notified. That being said, these messages are not usually a concern for end-users, however -- they are typically more useful for the OMPI developers. For example, if a developer accidentally does something to make a plugin un-loadable (e.g., leaves a symbol out), having these messages displayed at mpirun time can be *very* useful. Plugins that are shipped in a tarball hopefully do not suffer from such issues :-), and usually have rpath information compiled in them so even LD_LIBRARY_PATH issues shouldn't be much of a problem. -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Patrick Jessee Sent: Wednesday, June 28, 2006 5:18 PM To: Open MPI Users Subject: [OMPI users] error messages for btl components that aren't loaded Hello. I'm getting some odd error messages in certain situations associated with the btl components (happens with both 1.0.2 and 1.1). When certain btl components are NOT loaded, openMPI issues error messages associated with those very components. For instance, consider an application that is built with an openMPI installation that was configured with mvapi and mx (in addition to tcp,sm,self). If that application is taken to a system that does not have mvapi and mx interconnects installed and is explicitly started for TCP by using "--mca btl self,tcp,sm", then the following comes from openMPI: [devi01:01659] mca: base: component_find: unable to open: libvapi.so: cannot open shared object file: No such file or directory (ignored) [devi01:01659] mca: base: component_find: unable to open: libvapi.so: cannot open shared object file: No such file or directory (ignored) [devi01:01659] mca: base: component_find: unable to open: libmyriexpress.so: cannot open shared object file: No such file or directory (ignored) [devi02:31845] mca: base: component_find: unable to open: libvapi.so: cannot open shared object file: No such file or directory (ignored) [devi02:31845] mca: base: component_find: unable to open: libvapi.so: cannot open shared object file: No such file or directory (ignored) [devi02:31845] mca: base: component_find: unable to open: libmyriexpress.so: cannot open shared object file: No such file or directory (ignored) These are not fatal, but they definitely give the wrong impression that something is not right. The "--mca btl self,tcp,sm" op
Re: [OMPI users] MPI_Bcast/MPI_Finalize hang with Open MPI 1.1
On Jun 29, 2006, at 11:16 PM, Graham E Fagg wrote: On Thu, 29 Jun 2006, Doug Gregor wrote: When I use algorithm 6, I get: [odin003.cs.indiana.edu:14174] *** An error occurred in MPI_Bcast [odin005.cs.indiana.edu:10510] *** An error occurred in MPI_Bcast Broadcasting integers from root 0...[odin004.cs.indiana.edu:11752] *** An error occurred in MPI_Bcast ops.. my mistake!.. only 0-5 are valid for bcast I have to change the error message: [reliant:06935] coll:tuned:bcast_intra_do_forced algorithm 6 [reliant:06935] coll:tuned:bcast_intra_do_forced attempt to select algorithm 6 when only 0-6 is valid? I'm still trying to find out why it hangs.. let you know as soon as I find anything but right now I am testing using TCP. FWIW, I was able to reproduce the problem with both tcp and mvapi. Can you let me know the exact path and LD_LIBRARY_PATH your using on odin? PATH=/san/mpi/openmpi-1.1-gcc/bin:/u/dgregor/bin:/usr/kerberos/bin:/ usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/san/intel/cc/9.0/bin:/san/ intel/fc/9.0/bin:/san/intel/idb/9.0/bin:/san/pathscale/bin:/usr/local/ sched/slurm/bin LD_LIBRARY_PATH=/san/mpi/openmpi-1.1-gcc/lib:/san/intel/cc/9.0/lib: Doug
[OMPI users] Datatype bug regression from Open MPI 1.0.2 to Open MPI 1.1
Hello,
I had encountered a bug in Open MPI 1.0.1 using indexed datatypes
with MPI_Recv (which seems to be of the "off by one" sort), which
was corrected in Open MPI 1.0.2.
It seems to have resurfaced in Open MPI 1.1 (I encountered it using
different data and did not recognize it immediately, but it seems
it can reproduced using the same simplified test I had sent
the first time, which I re-attach here just in case).
Here is a summary of the case:
--
Each processor reads a file ("data_p0" or "data_p1") giving a list of
global element ids. Some elements (vertices from a partitionned mesh)
may belong to both processors, so their id's may appear on both
processors: we have 7178 global vertices, 3654 and 3688 of them being
known by ranks 0 and 1 respectively.
In this simplified version, we assign coordinates {x, y, z} to each
vertex equal to it's global id number for rank 1, and the negative of
that for rank 0 (assigning the same values to x, y, and z). After
finishing the "ordered gather", rank 0 prints the global id and
coordinates of each vertex.
lines should print (for example) as:
6456 ; 6455.0 6455.0 6456.0
6457 ; -6457.0 -6457.0 -6457.0
depending on whether a vertex belongs only to rank 0 (negative
coordinates) or belongs to rank 1 (positive coordinates).
With the OMPI 1.0.1 bug (observed on Suse Linux 10.0 with gcc 4.0 and on
Debian sarge with gcc 3.4), we have for example for the last vertices:
7176 ; 7175.0 7175.0 7176.0
7177 ; 7176.0 7176.0 7177.0
seeming to indicate an "off by one" type bug in datatype handling
Not using an indexed datatype (i.e. not defining USE_INDEXED_DATATYPE
in the gather_test.c file), the bug dissapears.
--
Best regards,
Yvan Fournier
ompi_datatype_bug.tar.gz
Description: application/compressed-tar
Re: [OMPI users] Testing one-sided message passing with 1.1
On Jun 29, 2006, at 5:23 PM, Tom Rosmond wrote: I am testing the one-sided message passing (mpi_put, mpi_get) that is now supported in the 1.1 release. It seems to work OK for some simple test codes, but when I run my big application, it fails. This application is a large weather model that runs operationally on the SGI Origin 3000, using the native one-sided message passing that has been supported on that system for many years. At least on that architecture, the code always runs correctly for processor numbers up to 480. On the O3K a requirement for the one-sided communication to work correctly is to use 'mpi_win_create' to define the RMA 'windows' in symmetric locations on all processors, i.e. the same 'place' in memory on each processor. This can be done with static memory, i.e. , in common; or on the 'symmetric heap', which is defined via environment variables. In my application the latter method is used. I define several of these 'windows' on the symmetric heap, each with a unique handle. Before I spend my time trying to diagnose this problem further, I need as much information about the OpenMPI one-sided implementation as available. Do you have a similar requirement or criteria for symmetric memory for the RMA windows? Are there runtime parameters that I should be using that are unique to one-sided message passing with OpenMPI? Any other information will certainly be appreciated. There are no requirements on the one-sided windows in terms of buffer pointers. Our current implementation is over point-to-point so it's kinda slow compared to real one-sided implementations, but has the advantage of working with arbitrary window locations. There is only two parameters to tweak in the current implementation: osc_pt2pt_eager_send: If this is 1, we try to start progressing the put/get before the synchronization point. The default is 0. This is not well tested, so I recommend leaving it 0. It's safer at this point. osc_pt2pt_fence_sync_method: This one might be worth playing with, but I doubt it could cause your problems. This is the collective we use to implement MPI_FENCE. Options are reduce_scatter (default), allreduce, alltoall. Again, I doubt it will make any difference, but would be interesting to confirm that. You can set the parameters at mpirun time: mpirun -np XX -mca osc_pt2pt_fence_sync_method reduce_scatter ./ test_code Our one-sided implementation has not been as well tested as the rest of the code (as this is our first release with one-sided support). If you can share any details on your application or, better yet, a test case, we'd appreciate it. There is one known issue with the implementation. It does not support using MPI_ACCUMULATE with user-defined datatypes, even if they are entirely composed of one predefined datatype. We plan on fixing this in the near future, and an error message will be printed if this situation occurs. Brian -- Brian Barrett Open MPI developer http://www.open-mpi.org/
