Re: [OMPI users] Displaying Selected MCA Modules
On Jun 20, 2008, at 3:50 PM, Joshua Bernstein wrote: No, we don't have an easy way to show which plugins were loaded and may/will be used during the run. The modules you found below in -- display-map are only a few of the plugins (all dealing with the run- time environment, and only used on the back-end nodes, so it may not be what you're looking for -- e.g., it doesn't show the plugins used by mpirun). What do you need to know? Well basically I want to know what MTA's are being used to startup a job. MTA? I'm confused as to what the difference is between "used by mpirun" versus user on the back-end nodes. Doesn't --display-map show which MTA modules will used to start the backend processes? Yes. But OMPI's run-time design usually has mpirun load one plugin of a given type, and then have the MPI processes load another plugin of the same type. For example, for I/O forwarding - mpirun will load the "svc" plugin, while MPI processes will load the "proxy" plugin. In this case, mpirun is actually providing all the smarts for I/O forwarding, and all the MPI processes simply proxy requests up to mpirun. This is a common model throughout our run-time support, for example. The overarching issue is that I'm attempting to just begin testing my build and when I attempt to startup a job, it just hangs: [ats@nt147 ~]$ mpirun --mca pls rsh -np 1 ./cpi [nt147.penguincomputing.com:04640] [0,0,0] ORTE_ERROR_LOG: Not available in file ras_bjs.c at line 247 The same thing happens if I just disable the bjs RAS MTA, since bjs, really isn't used with Scyld anymore: [ats@nt147 ~]$ mpirun --mca ras ^bjs --mca pls rsh -np 1 ./cpi I know very, very little about the bproc support in OMPI -- I know that it evolved over time and is disappearing in v1.3 due to lack of interest. If you want it to stay, I think you've missed the v1.3 boat (we're in feature freeze for v1.3), but possibilities exist for future versions if you're willing to get involved in Open MPI. The interesting thing here is that orted starts up, but I'm not sure what is supposed to happen next: [root@nt147 ~]# ps -auxwww | grep orte Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/ procps-3.2.3/FAQ ats 4647 0.0 0.0 48204 2136 ?Ss 12:45 0:00 orted --bootproxy 1 --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename nt147.penguincomputing.com --universe a...@nt147.penguincomputing.com:default-universe-4645 --nsreplica "0.0.0;tcp://192.168.5.211:59110;tcp://10.10.10.1:59110;tcp:// 10.11.10.1:59110" --gprreplica "0.0.0;tcp:// 192.168.5.211:59110;tcp://10.10.10.1:59110;tcp://10.11.10.1:59110" -- set-sid I'm not sure that just asking for the rsh pls is the Right thing to do -- I'll have to defer to Ralph on this one... Can you successfully run non-MPI apps, like hostname? Finally, it should be noted that the upcoming release of Scyld will now include OpenMPI. This notion is how all of this got started. Great! It sounds like you need to get involved, though, to preserve bproc support going forward. LANL was the only proponent of bproc- like support; they have been moving away from bproc-like clusters, however, and so support faded. We made the decision to axe bproc support in v1.3 because there was no one to maintain it. :-( -- Jeff Squyres Cisco Systems
Re: [OMPI users] parallel AMBER & PBS issue
Sorry for the delay in replying -- I was on vacation for a week and all the mail piled up... That is a very weird stack trace. Is the application finishing and then crashing during the shutdown? I'd be surprised if the problem is actually related to PBS (the stack trace would be quite different). I wonder if the real problem was that it only started one process, and Amber was unable to handle that nicely...? Are you sure that you have PBS support compiled in Open MPI properly? Check ompi_info | grep tm. You should see a line like this: MCA pls: tm (MCA v1.0, API v1.0.1, Component v1.2.6) If you don't see a "pls: tm" line, then your OMPI was not configured with PBS support, and mpiexec may have only started one copy of Amber...? As for trying to use a hostfile, I think the real errors are here: Host key verification failed. Host key verification failed. It seems that you ssh is not setup properly...? On Jun 12, 2008, at 11:52 AM, Arturas Ziemys wrote: Hi, We have Xeon dual cpu cluster on redhat. I have compiled openMPI 1.2.6 with g95 and AMBER (scientific program doing parallel molecular simulations; Fortran 77&90). Both compilation seems to be fine. However, AMBER runs from command prompt "mpiexec -np x " successfully, but using PBS batch system fails to run in parallel and runs only using single CPU. I get errors like: [Morpheus06:02155] *** Process received signal *** [Morpheus06:02155] Signal: Segmentation fault (11) [Morpheus06:02155] Signal code: Address not mapped (1) [Morpheus06:02155] Failing at address: 0x3900 [Morpheus06:02155] [ 0] /lib/tls/libpthread.so.0 [0x401ad610] [Morpheus06:02155] [ 1] /lib/tls/libc.so.6 [0x420eb85e] [Morpheus06:02155] [ 2] /lib/tls/libc.so.6(__cxa_finalize+0x7e) [0x42029eae] [Morpheus06:02155] [ 3] /home/aziemys/bin/openmpi/lib/libmpi_f90.so.0 [0x40018325] [Morpheus06:02155] [ 4] /home/aziemys/bin/openmpi/lib/libmpi_f90.so.0 [0x400190f6] [Morpheus06:02155] [ 5] /lib/ld-linux.so.2 [0x4000c894] [Morpheus06:02155] [ 6] /lib/tls/libc.so.6(exit+0x70) [0x42029c20] [Morpheus06:02155] [ 7] /home/aziemys/bin/amber9/exe/sander.MPI [0x82beb63] [Morpheus06:02155] [ 8] /home/aziemys/bin/amber9/exe/sander.MPI(_g95_exit_4+0x2c) [0x82bd648] [Morpheus06:02155] [ 9] /home/aziemys/bin/amber9/exe/sander.MPI(mexit_+0x9f) [0x817cd03] [Morpheus06:02155] [10] /home/aziemys/bin/amber9/exe/sander.MPI(MAIN_+0x3639) [0x80e8e51] [Morpheus06:02155] [11] /home/aziemys/bin/amber9/exe/sander.MPI(main+0x2d) [0x82bb471] [Morpheus06:02155] [12] /lib/tls/libc.so.6(__libc_start_main+0xe4) [0x42015574] [Morpheus06:02155] [13] /home/aziemys/bin/amber9/exe/sander.MPI(sinh+0x49) [0x80697a1] [Morpheus06:02155] *** End of error message *** mpiexec noticed that job rank 0 with PID 2150 on node Morpheus06 exited on signal 11 (Segmentation fault). 5 additional processes aborted (not shown) If I decide to supply machine file ($PBS_NODEFILE), it fails with : Host key verification failed. Host key verification failed. [Morpheus06:02107] ERROR: A daemon on node Morpheus09 failed to start as expected. [Morpheus06:02107] ERROR: There may be more information available from [Morpheus06:02107] ERROR: the remote shell (see above). [Morpheus06:02107] ERROR: The daemon exited unexpectedly with status 255. [Morpheus06:02107] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275 [Morpheus06:02107] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1166 [Morpheus06:02107] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90 [Morpheus06:02107] ERROR: A daemon on node Morpheus07 failed to start as expected. [Morpheus06:02107] ERROR: There may be more information available from [Morpheus06:02107] ERROR: the remote shell (see above). [Morpheus06:02107] ERROR: The daemon exited unexpectedly with status 255. [Morpheus06:02107] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 188 [Morpheus06:02107] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1198 -- mpiexec was unable to cleanly terminate the daemons for this job. Returned value Timeout instead of ORTE_SUCCESS. -- Help, please. -- Arturas Ziemys, PhD School of Health Information Sciences University of Texas Health Science Center at Houston 7000 Fannin, Suit 880 Houston, TX 77030 Phone: (713) 500-3975 Fax: (713) 500-3929 ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] Problem with getting started [solved]
It could also have been that you didn't have exactly matching
installations on both machines. Even if they were the same version,
if they weren't configured / installed the same way on both machines,
this could have led to problems. Also be sure that either the MPI
application is compatible / runnable on both systems or you have a
separate MPI application binary for each system (e.g., to account for
glibc and other differences between your two OS's).
Running in heterogeneous situations like that is quite difficult to
do, and not for the meek. :-)
On Jun 13, 2008, at 2:12 AM, Manuel Freiberger wrote:
Hello,
Well, actually I'm quite sure that it was not the firewall because I
had to
turn it off as otherwise no connection could be established. So my
iptables --list
returns
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
on both machines. After reinstalling OMPI, I did not make any
changes to the
firewall but now it works without problems. Probably installing the
library
with exactly the same configuration (same --prefix and so on) did
the trick.
But nonetheless, thank you very much for your hint! :-)
Best regards,
Manuel
On Thursday 12 June 2008 18:23, Rainer Keller wrote:
Hi,
are You sure it was not a Firewall issue on the Suse 10.2?
If there are any connections from the Gentoo machine trying to
access the
orted on the Suse, check in /var/log/firewall.
For the time being, try stopping the firewall by (as root) with
/etc/init.d/SuSEfirewall2_setup stop
and test whether it works ,-]
With best regards,
Rainer
On Donnerstag, 12. Juni 2008, Manuel Freiberger wrote:
Hi!
Ok, I found the problem. I reinstallen OMPI on both PCs but this
time
only locally in the users home directory. Now, the sample code works
perfectly. I'm not sure where the error really was located. It
could be
that it was a problem with the Gentoo installation because OMPI is
still
marked unstable in portage (~x86 keyword).
Best regards,
Manuel
On Wednesday 11 June 2008 18:52, Manuel Freiberger wrote:
Hello everybody!
First of all I wanted to point out that I'm beginner regarding
openMPI
and all I try to achieve is to get a simple program working on
two PCs.
So far I've installed openMPI 1.2.6 on two PCs (one running
OpenSUSE
10.2, the other one Gentoo).
I set up two identical users on both systems and made sure that I
can
make an SSH connection between them using private/public key
authentication.
Next I ran the command
mpirun -np 2 --hostfile myhosts uptime
which gave the result
6:41pm up 1 day 5:16, 4 users, load average: 0.00, 0.07, 0.17
18:43:45 up 7:36, 6 users, load average: 0.00, 0.02, 0.05
so I concluded that MPI should work in principle.
Next I tried the following code which I copied from Boost.MPI:
snip
#include
#include
int main(int argc, char* argv[])
{
MPI_Init(&argc, &argv);
int rank;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0)
{
std::cout << "Rank 0 is going to send" << std::endl;
int value = 17;
int result = MPI_Send(&value, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
if (result == MPI_SUCCESS)
std::cout << "Rank 0 OK!" << std::endl;
}
else if (rank == 1)
{
std::cout << "Rank 1 is waiting for answer" << std::endl;
int value;
MPI_Status status;
int result = MPI_Recv(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD,
&status);
if (result == MPI_SUCCESS && value == 17)
std::cout << "Rank 1 OK!" << std::endl;
}
MPI_Finalize();
return 0;
}
snap
Starting a parallel job using
mpirun -np 2 --hostfile myhosts mpi-test
I get the answer
Rank 0 is going to send
Rank 1 is waiting for answer
Rank 0 OK!
and than the program locks. So the strange thing is that
obviously the
recv()-command is blocking, which is what I do not understand.
Could anybody provide some hints, where I should start looking
for the
mistake? Any help is welcome!
Best regards,
Manuel
--
Manuel Freiberger
Institute of Medical Engineering
Graz University of Technology
manuel.freiber...@tugraz.at
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
Re: [OMPI users] weird problem with passing data between nodes
Yes, this does sound like the classic "assuming MPI buffering" case. Check out this magazine column that I wrote a long time ago about this topic: http://cw.squyres.com/columns/2004-08-CW-MPI-Mechanic.pdf It's #1 on the top 10 list of All-Time Favorite Evils to Avoid in Parallel. :-) One comment on Mattijs's email: please don't use bsend. Bsend is evil. :-) On Jun 13, 2008, at 5:27 AM, Mattijs Janssens wrote: Sounds like a typical deadlock situation. All processors are waiting for one another. Not a specialist but from what I know if the messages are small enough they'll be offloaded to kernel/hardware and there is no deadlock. That why it might work for small messages and/or certain mpi implementations. Solutions: - come up with a global communication schedule such that if one processor sends the receiver is receiving. - use mpi_bsend. Might be slower. - use mpi_isend, mpi_irecv (but then you'll have to make sure the buffers stay valid for the duration of the communication) On Friday 13 June 2008 01:55, zach wrote: I have a weird problem that shows up when i use LAM or OpenMPI but not MPICH. I have a parallelized code working on a really large matrix. It partitions the matrix column-wise and ships them off to processors, so, any given processor is working on a matrix with the same number of rows as the original but reduced number of columns. Each processor needs to send a single column vector entry from its own matrix to the adjacent processor and visa versa as part of the algorithm. I have found that depending on the number of rows of the matrix -or, the size of the vector being sent using MPI_Send, MPI_Recv, the simulation will hang. It is only until i reduce this dimension to a certain max number will the sim run properly. I have also found that this magic number differs depending on the system I am using, eg my home quad-core box or remote cluster. As i mentioned i have not had this issue with mpich. I would like to understand why it is happening rather than just defect over to mpich to get by. Any help would be appreciated! zach ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] mpirun (orte ?) not shutting down cleanly on job aborts
Sorry for the delay in replying to this -- mails sometimes pile up in my INBOX and I don't get to reply to them all in a timely fashion. Yes, you can expect this to be much better in the v1.3 series. If you have a few cycles, you might want to test a nightly trunk tarball snapshot in some of your problematic cases and see if it's better. We've had a little instability in trunk tarballs over the last week, so you might want to wait until next week to give it a shot. http://www.open-mpi.org/nightly/trunk/ On Jun 9, 2008, at 10:50 AM, Bill Johnstone wrote: Hello OMPI devs, I'm currently running OMPI v 1.2.4 . It didn't seem that any bugs which affect me or my users were fixed in 1.2.5 and 1.2.6, so I haven't upgraded yet. When I was initially getting started with OpenMPI, I had some problems which I was able to solve, but one still remains. As I mentioned in http://www.open-mpi.org/community/lists/users/2007/07/3716.php when there is a non-graceful exit on any of the MPI jobs, mpirun hangs. As an example, I have a code that I run which, when it has a trivial runtime error (e.g., some small mistake in the input file) dies yielding messages to the screen like: [node1.x86-64:28556] MPI_ABORT invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 16 but mpirun never exits, and Ctrl+C won't kill it. I have to resort to kill -9. Now that I'm running under SLURM, this is worse because there is no nice way to manually clear individual jobs off the controller. So even if I manually kill mpirun on the failed job, slurmctld still thinks its running. Ralph Castain replied to the previously-linked message: http://www.open-mpi.org/community/lists/users/2007/07/3718.php indicating that he thought he knew why this was happening and that it was or would likely be fixed in the trunk. At this point, I just want to know: can I look forward to this being fixed in the upcoming v 1.3 series? I don't mean that to sound ungrateful: *many thanks* to the OMPI devs for what you've already given the community at large. I'm just a bit frustrated because we seem to run a lot of codes on our cluster that abort at one time or another. Thank you. ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] no reaction of remote hosts after ompi reinstall [follow up]
Sorry for the delay in replying. I'd check two things: - Disable all firewall support between these two machines. OMPI uses random TCP ports to communicate between processes; if they're blocked, Bad Things will happen. - It is easiest to install OMPI in the same location on all your machines (e.g., /opt/openmpi). If you do that, you might want to try configuring OMPI with --enable-mpirun-prefix-by-default. In rsh/ssh environments, this flag will have mpirun set your PATH and LD_LIBRARY_PATH properly on remote nodes. Let us know how that works out. On Jun 10, 2008, at 8:58 AM, jody wrote: Interestingly i can start mpirun from any of the remote machines, running processes on other remote machines and on the local machine,. But from the local machine i can start no process on a remote machine - it just shows the behavior detailed in the previous mail. remote1 -> remote1 ok remote1 -> remote2 ok remote1 -> local ok remote2 -> remote1 ok remote2 -> remote2 ok remote2 -> local ok local -> local ok local -> remote1 fails local -> remote2 fails My remote machines are freshly updated gentoo machines (AMD), my local machne is a freshly installed fedora 8 (Intel Quadro). All use a freshly installed open-mpi 1.2.5. Before my fedora machine crashed it had fedora 6, and everything worked great (with 1.2.2 on all machines). Does anybody have a suggestion where i should look? Thanks Jody On Tue, Jun 10, 2008 at 12:59 PM, jody wrote: Hi after a crash i reinstalled open-mpi 1.2.5 on my machines, used ./configure --prefix /opt/openmpi --enable-mpirun-prefix-by-default and set PATH and LD_LIBRARY_PATH in .bashrc: PATH=/opt/openmpi/bin:$PATH export PATH LD_LIBRARY_PATH=/opt/openmpi/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH First problem: ssh nano_00 printenv does not contain the correct paths (and no LD_LIBRARY_PATH at all), but with a normal ssh-login the two are set correctly. When i run a test application on one computer, it works. As soon as an additional computer is involved, there is no more output, and everything just hangs. Adding the prefix doesn't change anything, even though openmpi is installed in the same directory (/opt/openmpi) on every computer. The debug-daemon doesn't help very much: $ mpirun -np 4 --hostfile testhosts --debug-daemons MPITest Daemon [0,0,1] checking in as pid 14927 on host aim-plankton.uzh.ch (and nothing happens anymore) On the remote host, i see the following three processes coming up after i do the mpirun on the local machine: 30603 ?S 0:00 sshd: jody@notty 30604 ?Ss 0:00 bash -c PATH=/opt/openmpi/bin:$PATH ; export PATH ; LD_LIBRARY_PATH=/opt/openmpi/lib:$LD_LIBRARY_PATH ; export LD_LIBRARY_PATH ; /opt/openmpi/bin/orted --debug-daemons --bootproxy 1 --name 0.0.2 --num_procs 3 --vpid_start 0 -- 30605 ?S 0:00 /opt/openmpi/bin/orted --debug-daemons --bootproxy 1 --name 0.0.2 --num_procs 3 --vpid_start 0 --nodename nano_00 --universe j...@aim-plankton.uzh.ch:default-universe-14934 --nsreplica 0.0.0;tcp://130.60.126.111:52562 --gprrepl So it looks as if the correct paths are set (probably the doing of --enable-mpirun-prefix-by-default) If i interrupt on the local machine (Ctrl-C):: [aim-plankton:14983] [0,0,1] orted_recv_pls: received message from [0,0,0] [aim-plankton:14983] [0,0,1] orted_recv_pls: received kill_local_procs [aim-plankton:14983] [0,0,1] orted_recv_pls: received message from [0,0,0] [aim-plankton:14983] [0,0,1] orted_recv_pls: received kill_local_procs [aim-plankton:14982] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275 [aim-plankton:14982] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1166 [aim-plankton:14982] [0,0,0] ORTE_ERROR_LOG: Timeout in file errmgr_hnp.c at line 90 [aim-plankton:14982] ERROR: A daemon on node nano_00 failed to start as expected. [aim-plankton:14982] ERROR: There may be more information available from [aim-plankton:14982] ERROR: the remote shell (see above). [aim-plankton:14982] ERROR: The daemon exited unexpectedly with status 255. [aim-plankton:14982] [0,0,0] ORTE_ERROR_LOG: Timeout in file base/pls_base_orted_cmds.c at line 275 [aim-plankton:14982] [0,0,0] ORTE_ERROR_LOG: Timeout in file pls_rsh_module.c at line 1166 -- WARNING: mpirun has exited before it received notification that all started processes had terminated. You should double check and ensure that there are no runaway processes still executing. -- [aim-plankton:14983] OOB: Connection to HNP lost On the remote machine, the "sshd: jody@notty" process is gone, but the other two stay. I would be grateful for any suggestions! Jody ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cg
Re: [OMPI users] OpenIB problem: error polling HP CQ...
Errr... That's not good. :-( Do you have a small example that you can share that duplicates the problem? On Jun 6, 2008, at 1:51 AM, Matt Hughes wrote: 2008/6/4 Jeff Squyres : Would it be possible for you to try a trunk nightly tarball snapshot, perchance? I have attempted to use openmpi-1.3a1r18569. After some pain getting MPI_Comm_spawn to work (I will write about that in a separate message), I was able to get my app started. If segfaults in btl_openib_handle_incoming() by dereferencing a null pointer: #0 0x in ?? () #1 0x002a98059777 in btl_openib_handle_incoming (openib_btl=0xb8b900, ep=0xbecb70, frag=0xc8da80, byte_len=24) at btl_openib_component.c: 2129 #2 0x002a9805b674 in handle_wc (hca=0xb80670, cq=0, wc=0x7fbfffdfd0) at btl_openib_component.c:2397 #3 0x002a9805bbef in poll_hca (hca=0xb80670, count=1) at btl_openib_component.c:2508 #4 0x002a9805c1ac in progress_one_hca (hca=0xb80670) at btl_openib_component.c:2616 #5 0x002a9805c24f in btl_openib_component_progress () at btl_openib_component.c:2641 #6 0x002a97f42308 in mca_bml_r2_progress () at bml_r2.c:93 #7 0x002a95a44c2c in opal_progress () at runtime/ opal_progress.c:187 #8 0x002a97d1f10c in opal_condition_wait (c=0x2a958b8b40, m=0x2a958b8bc0) at ../../../../opal/threads/condition.h:100 #9 0x002a97d1ef88 in ompi_request_wait_completion (req=0xbdfc80) at ../../../../ompi/request/request.h:381 #10 0x002a97d1ee64 in mca_pml_ob1_recv (addr=0xc52d14, count=1, datatype=0x2a958abe60, src=1, tag=-19, comm=0xbe0cf0, status=0x0) at pml_ob1_irecv.c:104 #11 0x002a98c1b182 in ompi_coll_tuned_gather_intra_basic_linear ( sbuf=0x7fbfffe984, scount=1, sdtype=0x2a958abe60, rbuf=0xc52d10, rcount=1, rdtype=0x2a958abe60, root=0, comm=0xbe0cf0, module=0xda00e0) at coll_tuned_gather.c:408 #12 0x002a98c07fc1 in ompi_coll_tuned_gather_intra_dec_fixed ( sbuf=0x7fbfffe984, scount=1, sdtype=0x2a958abe60, rbuf=0xc52d10, rcount=1, rdtype=0x2a958abe60, root=0, comm=0xbe0cf0, module=0xda00e0) at coll_tuned_decision_fixed.c:723 #13 0x002a95715f0f in PMPI_Gather (sendbuf=0x7fbfffe984, sendcount=1, sendtype=0x2a958abe60, recvbuf=0xc52d10, recvcount=1, recvtype=0x2a958abe60, root=0, comm=0xbe0cf0) at pgather.c:141 This same build works fine with the TCP component and at least doesn't crash with 1.2.6. The only thing that may be unusual about my build of openmpi 1.3 is that it is configured with --without-memory-manager (it seems to cause crashes in another library I use). I tried rebuilding, omitting --without-memory-manager, but it failed in the same way. mch On May 29, 2008, at 3:50 AM, Matt Hughes wrote: I have a program which uses MPI::Comm::Spawn to start processes on compute nodes (c0-0, c0-1, etc). The communication between the compute nodes consists of ISend and IRecv pairs, while communication between the compute nodes consists of gather and bcast operations. After executing ~80 successful loops (gather/bcast pairs), I get this error message from the head node process during a gather call: [0,1,0][btl_openib_component.c:1332:btl_openib_component_progress] from headnode.local to: c0-0 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 18504944 opcode 1 The relevant environment variables: OMPI_MCA_btl_openib_rd_num=128 OMPI_MCA_btl_openib_verbose=1 OMPI_MCA_btl_base_verbose=1 OMPI_MCA_btl_openib_rd_low=75 OMPI_MCA_btl_base_debug=1 OMPI_MCA_btl_openib_warn_no_hca_params_found=0 OMPI_MCA_btl_openib_warn_default_gid_prefix=0 OMPI_MCA_btl=self,openib If rd_low and rd_num are left at their default values, the program simply hangs in the gather call after about 20 iterations (a gather and a bcast). Can anyone shed any light on what this error message means or what might be done about it? Thanks, mch ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] Displaying Selected MCA Modules
Hmmmthis email thread was forwarded to me by a friendly elf, so be sure to include me directly in any reply. As the person who wrote a lot of the bproc support, I'll try to catch up on it over the next couple of days. One thing to correct right away, though. The --display-map function actually doesn't tell you what modules are going to be used to run the job. The values you show below are the MCA params being passed to the application procs, which have nothing to do with how those procs are being launched! In fact, in 1.2, we hard-code the "proxy" components solely to turn "off" any attempt by a local proc to mistakenly choose a "real" component for those frameworks. The "proxy" just tells them to send a message to mpirun if they actually need to use any of those frameworks. In a bproc world, NONE of those frameworks are used by the procs, and the procs never call any of them. I'm not sure that 1.2 provides a mechanism for reliably reporting what every framework selected. Ralph On 6/19/08 5:00 PM, "Joshua Bernstein" wrote: > Well to answer my own question, > > If I use the -display-map option, I get printed out a nice bit of > information that includes a list of the modules in use during the run > as shown below: > > ---SNIP--- > Argv[0]: ./cpi > Env[0]: OMPI_MCA_pls=proxy > Env[1]: OMPI_MCA_rmaps_base_display_map=1 > Env[2]: > OMPI_MCA_orte_precondition_transports=ad81e32181314110-4aea4dd5040c2593 > Env[3]: OMPI_MCA_rds=proxy > Env[4]: OMPI_MCA_ras=proxy > Env[5]: OMPI_MCA_rmaps=proxy > Env[6]: OMPI_MCA_rmgr=proxy > Working dir: /home/ats (user: 0) > ---END SNIP-- > > -Josh > > Joshua Bernstein wrote: >> Hi There, >>I'm attempting to debug some configuration issue with the recent >> version of OMPI, version 1.2.6. I'm able to build all of the MCA >> modules, and I've figured out how to display the list of AVAILABLE >> modules using ompi_info, but is there a way to display the list of >> modules that was selected at runtime? I've tried the -v option to >> mpirun, and read through the FAQs, but I can't seem to figure out >> how to have OMPI display the selected MCAs when a job starts. Any >> help or guidance would be appreciated. >> -Josh >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Displaying Selected MCA Modules
Hi Joshua Again, forwarded by the friendly elf - so include me directly in any reply. I gather from Jeff that you are attempting to do something with bproc - true? If so, I will echo what Jeff said: bproc support in OMPI is being dropped with the 1.3 release due to lack of interest/support. Just a "heads up". If you are operating in a bproc environment, then I'm not sure why you are specifying that the system use the rsh launcher. Bproc requires some very special handling which is only present in the bproc launcher. You can run both MPI and non-MPI apps with it, but bproc is weird and so OMPI some -very- different logic in it to make it all work. I suspect the problem you are having is that all of the frameworks are detecting bproc and trying to run accordingly. This means that the orted is executing process startup procedures for bproc - which are totally different than for any other environment (e.g., rsh). If mpirun is attempting to execute an rsh launch, and the orted is expecting a bproc launch, then I can guarantee that no processes will be launched and you will hang. I'm not sure there is a way in 1.2 to tell the orteds to ignore the fact that they see bproc and do something else. I can look, but would rather wait to hear if that is truly what you are trying to do, and why. Ralph On 6/21/08 5:10 AM, "Jeff Squyres" wrote: > On Jun 20, 2008, at 3:50 PM, Joshua Bernstein wrote: > >>> No, we don't have an easy way to show which plugins were loaded and >>> may/will be used during the run. The modules you found below in -- >>> display-map are only a few of the plugins (all dealing with the run- >>> time environment, and only used on the back-end nodes, so it may >>> not be what you're looking for -- e.g., it doesn't show the plugins >>> used by mpirun). >>> What do you need to know? >> >> Well basically I want to know what MTA's are being used to startup a >> job. > > MTA? > >> I'm confused as to what the difference is between "used by mpirun" >> versus user on the back-end nodes. Doesn't --display-map show which >> MTA modules will used to start the backend processes? > > Yes. But OMPI's run-time design usually has mpirun load one plugin of > a given type, and then have the MPI processes load another plugin of > the same type. For example, for I/O forwarding - mpirun will load the > "svc" plugin, while MPI processes will load the "proxy" plugin. In > this case, mpirun is actually providing all the smarts for I/O > forwarding, and all the MPI processes simply proxy requests up to > mpirun. This is a common model throughout our run-time support, for > example. > >> The overarching issue is that I'm attempting to just begin testing >> my build and when I attempt to startup a job, it just hangs: >> >> [ats@nt147 ~]$ mpirun --mca pls rsh -np 1 ./cpi >> [nt147.penguincomputing.com:04640] [0,0,0] ORTE_ERROR_LOG: Not >> available in file ras_bjs.c at line 247 >> >> The same thing happens if I just disable the bjs RAS MTA, since bjs, >> really isn't used with Scyld anymore: >> >> [ats@nt147 ~]$ mpirun --mca ras ^bjs --mca pls rsh -np 1 ./cpi >> > > I know very, very little about the bproc support in OMPI -- I know > that it evolved over time and is disappearing in v1.3 due to lack of > interest. If you want it to stay, I think you've missed the v1.3 boat > (we're in feature freeze for v1.3), but possibilities exist for future > versions if you're willing to get involved in Open MPI. > >> The interesting thing here is that orted starts up, but I'm not sure >> what is supposed to happen next: >> >> [root@nt147 ~]# ps -auxwww | grep orte >> Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/ >> procps-3.2.3/FAQ >> ats 4647 0.0 0.0 48204 2136 ?Ss 12:45 0:00 orted >> --bootproxy 1 --name 0.0.1 --num_procs 2 --vpid_start 0 --nodename >> nt147.penguincomputing.com --universe >> a...@nt147.penguincomputing.com:default-universe-4645 --nsreplica >> "0.0.0;tcp://192.168.5.211:59110;tcp://10.10.10.1:59110;tcp:// >> 10.11.10.1:59110" --gprreplica "0.0.0;tcp:// >> 192.168.5.211:59110;tcp://10.10.10.1:59110;tcp://10.11.10.1:59110" -- >> set-sid > > I'm not sure that just asking for the rsh pls is the Right thing to do > -- I'll have to defer to Ralph on this one... > > Can you successfully run non-MPI apps, like hostname? > >> Finally, it should be noted that the upcoming release of Scyld will >> now include OpenMPI. This notion is how all of this got started. > > > Great! It sounds like you need to get involved, though, to preserve > bproc support going forward. LANL was the only proponent of bproc- > like support; they have been moving away from bproc-like clusters, > however, and so support faded. We made the decision to axe bproc > support in v1.3 because there was no one to maintain it. :-(
