On 08/08/11 23:46, Christopher Jones wrote:
Hi again,I changed the subject of my previous posting to reflect a new problem encountered when I changed my strategy to using SSH instead of Xgrid on two mac pros. I've set up a login-less ssh communication between the two macs (connected via direct ethernet, both running openmpi 1.2.8 on OSX 10.6.8) per the instructions on the FAQ. I can type in 'ssh computer-name.local' on either computer and connect without a password prompt. From what I can see, the ssh-agent is up and running - the following is listed in my ENV: SSH_AUTH_SOCK=/tmp/launch-5FoCc1/Listeners SSH_AGENT_PID=61058 My host file simply lists 'localhost' and 'chrisjones2@allana-welshs-mac-pro.local'. When I run a simple hello_world test, I get what seems like a reasonable output: chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile ./test_hello Hello world from process 0 of 8 Hello world from process 1 of 8 Hello world from process 2 of 8 Hello world from process 3 of 8 Hello world from process 4 of 8 Hello world from process 7 of 8 Hello world from process 5 of 8 Hello world from process 6 of 8 I can also run hostname and get what seems to be an ok response (unless I'm wrong about this): chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile hostname allana-welshs-mac-pro.local allana-welshs-mac-pro.local allana-welshs-mac-pro.local allana-welshs-mac-pro.local quadcore.mikrob.slu.se quadcore.mikrob.slu.se quadcore.mikrob.slu.se quadcore.mikrob.slu.se However, when I run the ring_c test, it freezes: chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile ./ring_c Process 0 sending 10 to 1, tag 201 (8 processes in ring) Process 0 sent to 1 Process 0 decremented value: 9 (I noted that processors on both computers are active). ring_c was compiled separately on each computer, however both have the same version of openmpi and OSX. I've gone through the FAQ and searched the user forum, but I can't quite seems to get this problem unstuck. Many thanks for your time, Chris On Aug 5, 2011, at 6:00 PM,<users-requ...@open-mpi.org> <users-requ...@open-mpi.org> wrote:Send users mailing list submissions to us...@open-mpi.org To subscribe or unsubscribe via the World Wide Web, visit http://www.open-mpi.org/mailman/listinfo.cgi/users or, via email, send a message with subject or body 'help' to users-requ...@open-mpi.org You can reach the person managing the list at users-ow...@open-mpi.org When replying, please edit your Subject line so it is more specific than "Re: Contents of users digest..." Today's Topics: 1. Re: OpenMPI causing WRF to crash (Jeff Squyres) 2. Re: OpenMPI causing WRF to crash (Anthony Chan) 3. Re: Program hangs on send when run with nodes on remote machine (Jeff Squyres) 4. Re: openmpi 1.2.8 on Xgrid noob issue (Jeff Squyres) 5. Re: parallel I/O on 64-bit indexed arays (Rob Latham) ---------------------------------------------------------------------- Message: 1 Date: Thu, 4 Aug 2011 19:18:36 -0400 From: Jeff Squyres<jsquy...@cisco.com> Subject: Re: [OMPI users] OpenMPI causing WRF to crash To: Open MPI Users<us...@open-mpi.org> Message-ID:<3f0e661f-a74f-4e51-86c0-1f84feb07...@cisco.com> Content-Type: text/plain; charset=windows-1252 Signal 15 is usually SIGTERM on Linux, meaning that some external entity probably killed the job. The OMPI error message you describe is also typical for that kind of scenario -- i.e., a process exited without calling MPI_Finalize could mean that it called exit() or some external process killed it. On Aug 3, 2011, at 7:24 AM, BasitAli Khan wrote:I am trying to run a rather heavy wrf simulation with spectral nudging but the simulation crashes after 1.8 minutes of integration. The simulation has two domains with d01 = 601x601 and d02 = 721x721 and 51 vertical levels. I tried this simulation on two different systems but result was more or less same. For example On our Bluegene/P with SUSE Linux Enterprise Server 10 ppc and XLF compiler I tried to run wrf on 2048 shared memory nodes (1 compute node = 4 cores , 32 bit, 850 Mhz). For the parallel run I used mpixlc, mpixlcxx and mpixlf90. I got the following error message in the wrf.err file <Aug 01 19:50:21.244540> BE_MPI (ERROR): The error message in the job record is as follows: <Aug 01 19:50:21.244657> BE_MPI (ERROR): "killed with signal 15" I also tried to run the same simulation on our linux cluster (Linux Red Hat Enterprise 5.4m x86_64 and Intel compiler) with 8, 16 and 64 nodes (1 compute node=8 cores). For the parallel run I am used mpi/openmpi/1.4.2-intel-11. I got the following error message in the error log after couple of minutes of integration. "mpirun has exited due to process rank 45 with PID 19540 on node ci118 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here)." I tried many things but nothing seems to be working. However, if I reduce grid points below 200, the simulation goes fine. It appears that probably OpenMP has problem with large number of grid points but I have no idea how to fix it. I will greatly appreciate if you could suggest some solution. Best regards, --- Basit A. Khan, Ph.D. Postdoctoral Fellow Division of Physical Sciences& Engineering Office# 3204, Level 3, Building 1, King Abdullah University of Science& Technology 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, Kingdom of Saudi Arabia. Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 E-mail: basitali.k...@kaust.edu.sa Skype name: basit.a.khan _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users-- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ------------------------------ Message: 2 Date: Thu, 4 Aug 2011 18:59:59 -0500 (CDT) From: Anthony Chan<c...@mcs.anl.gov> Subject: Re: [OMPI users] OpenMPI causing WRF to crash To: Open MPI Users<us...@open-mpi.org> Message-ID: <660521091.191111.1312502399225.javamail.r...@zimbra.anl.gov> Content-Type: text/plain; charset=utf-8 If you want to debug this on BGP, you could set BG_COREDUMPONERROR=1 and look at the backtrace in the light weight core files (you probably need to recompile everything with -g). A.Chan ----- Original Message -----Hi Dmitry, Thanks for a prompt and fairly detailed response. I have also forwarded the email to wrf community in the hope that somebody would have some straight forward solution. I will try to debug the error as suggested by you if I would not have much luck from the wrf forum. Cheers, --- Basit A. Khan, Ph.D. Postdoctoral Fellow Division of Physical Sciences& Engineering Office# 3204, Level 3, Building 1, King Abdullah University of Science& Technology 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, Kingdom of Saudi Arabia. Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 E-mail: basitali.k...@kaust.edu.sa Skype name: basit.a.khan On 8/3/11 2:46 PM, "Dmitry N. Mikushin"<maemar...@gmail.com> wrote:5 apparently means one of the WRF's MPI processes has been unexpectedly terminated, maybe by program decision. No matter, if it is OpenMPI-specifi_______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users------------------------------ Message: 3 Date: Thu, 4 Aug 2011 20:46:16 -0400 From: Jeff Squyres<jsquy...@cisco.com> Subject: Re: [OMPI users] Program hangs on send when run with nodes on remote machine To: Open MPI Users<us...@open-mpi.org> Message-ID:<f344f301-ad7b-4e83-b0df-a6e001072...@cisco.com> Content-Type: text/plain; charset=us-ascii I notice that in the worker, you have: eth2 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d4 inet addr:192.168.1.155 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fe77:c5d4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:9225846 errors:0 dropped:75175 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1336628768 (1.3 GB) TX bytes:552 (552.0 B) eth3 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d5 inet addr:192.168.1.156 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:fe77:c5d5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:26481809 errors:0 dropped:75059 overruns:0 frame:0 TX packets:18030236 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:70061260271 (70.0 GB) TX bytes:11844181778 (11.8 GB) Two different NICs are on the same subnet -- that doesn't seem like a good idea...? I think this topic has come up on the users list before, and, IIRC, the general consensus is "don't do that" because it's not clear as to which NIC Linux will actually send outgoing traffic across bound for the 192.168.1.x subnet. On Aug 4, 2011, at 1:59 PM, Keith Manville wrote:I am having trouble running my MPI program on multiple nodes. I can run multiple processes on a single node, and I can spawn processes on on remote nodes, but when I call Send from a remote node, the node never returns, even though there is an appropriate Recv waiting. I'm pretty sure this is an issue with my configuration, not my code. I've tried some other sample programs I found and had the same problem of hanging on a send from one host to another. Here's an in depth description: I wrote a quick test program where each process with rank> 1 sends an int to the master (rank 0), and the master receives until it gets something from every other process. My test program works fine when I run multiple processes on a single machine. either the local node: $ ./mpirun -n 4 ./mpi-test Hi I'm localhost:2 Hi I'm localhost:1 localhost:1 sending 11... localhost:2 sending 12... localhost:2 sent 12 localhost:1 sent 11 Hi I'm localhost:0 localhost:0 received 11 from 1 localhost:0 received 12 from 2 Hi I'm localhost:3 localhost:3 sending 13... localhost:3 sent 13 localhost:0 received 13 from 3 all workers checked in! or a remote one: $ ./mpirun -np 2 -host remotehost ./mpi-test Hi I'm remotehost:0 remotehost:0 received 11 from 1 all workers checked in! Hi I'm remotehost:1 remotehost:1 sending 11... remotehost:1 sent 11 But when I try to run the master locally and the worker(s) remotely (this is the way I am actually interested in running it), Send never returns and it hangs indefinitely. $ ./mpirun -np 2 -host localhost,remotehost ./mpi-test Hi I'm localhost:0 Hi I'm remotehost:1 remotehost:1 sending 11... Just to see if it would work, I tried spawning the master on the remotehost and the worker on the localhost. $ ./mpirun -np 2 -host remotehost,localhost ./mpi-test Hi I'm localhost:1 localhost:1 sending 11... localhost:1 sent 11 Hi I'm remotehost:0 remotehost:0 received 0 from 1 all workers checked in! It doesn't hang on Send, but the wrong value is received. Any idea what's going on? I've attached my code, my config.log, ifconfig output, and ompi_info output. Thanks, Keith <mpi.tgz>_______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users-- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ------------------------------ Message: 4 Date: Thu, 4 Aug 2011 20:48:30 -0400 From: Jeff Squyres<jsquy...@cisco.com> Subject: Re: [OMPI users] openmpi 1.2.8 on Xgrid noob issue To: Open MPI Users<us...@open-mpi.org> Message-ID:<c2ea7fd0-badb-4d05-851c-c444be26f...@cisco.com> Content-Type: text/plain; charset=us-ascii I'm afraid our Xgrid support has lagged, and Apple hasn't show much interest in MPI + Xgrid support -- much less HPC. :-\ Have you see the FAQ items about Xgrid? http://www.open-mpi.org/faq/?category=osx#xgrid-howto On Aug 4, 2011, at 4:16 AM, Christopher Jones wrote:Hi there, I'm currently trying to set up a small xgrid between two mac pros (a single quadcore and a 2 duo core), where both are directly connected via an ethernet cable. I've set up xgrid using the password authentication (rather than the kerberos), and from what I can tell in the Xgrid admin tool it seems to be working. However, once I try a simple hello world program, I get this error: chris-joness-mac-pro:~ chrisjones$ mpirun -np 4 ./test_hello mpirun noticed that job rank 0 with PID 381 on node xgrid-node-0 exited on signal 15 (Terminated). 1 additional process aborted (not shown) 2011-08-04 10:02:16.329 mpirun[350:903] *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '*** -[NSKVONotifying_XGConnection<0x1001325a0> finalize]: called when collecting not enabled' *** Call stack at first throw: ( 0 CoreFoundation 0x00007fff814237b4 __exceptionPreprocess + 180 1 libobjc.A.dylib 0x00007fff84fe8f03 objc_exception_throw + 45 2 CoreFoundation 0x00007fff8143e631 -[NSObject(NSObject) finalize] + 129 3 mca_pls_xgrid.so 0x00000001002a9ce3 -[PlsXGridClient dealloc] + 419 4 mca_pls_xgrid.so 0x00000001002a9837 orte_pls_xgrid_finalize + 40 5 libopen-rte.0.dylib 0x000000010002d0f9 orte_pls_base_close + 249 6 libopen-rte.0.dylib 0x0000000100012027 orte_system_finalize + 119 7 libopen-rte.0.dylib 0x000000010000e968 orte_finalize + 40 8 mpirun 0x00000001000011ff orterun + 2042 9 mpirun 0x0000000100000a03 main + 27 10 mpirun 0x00000001000009e0 start + 52 11 ??? 0x0000000000000004 0x0 + 4 ) terminate called after throwing an instance of 'NSException' [chris-joness-mac-pro:00350] *** Process received signal *** [chris-joness-mac-pro:00350] Signal: Abort trap (6) [chris-joness-mac-pro:00350] Signal code: (0) [chris-joness-mac-pro:00350] [ 0] 2 libSystem.B.dylib 0x00007fff81ca51ba _sigtramp + 26 [chris-joness-mac-pro:00350] [ 1] 3 ??? 0x00000001000cd400 0x0 + 4295808000 [chris-joness-mac-pro:00350] [ 2] 4 libstdc++.6.dylib 0x00007fff830965d2 __tcf_0 + 0 [chris-joness-mac-pro:00350] [ 3] 5 libobjc.A.dylib 0x00007fff84fecb39 _objc_terminate + 100 [chris-joness-mac-pro:00350] [ 4] 6 libstdc++.6.dylib 0x00007fff83094ae1 _ZN10__cxxabiv111__terminateEPFvvE + 11 [chris-joness-mac-pro:00350] [ 5] 7 libstdc++.6.dylib 0x00007fff83094b16 _ZN10__cxxabiv112__unexpectedEPFvvE + 0 [chris-joness-mac-pro:00350] [ 6] 8 libstdc++.6.dylib 0x00007fff83094bfc _ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception + 0 [chris-joness-mac-pro:00350] [ 7] 9 libobjc.A.dylib 0x00007fff84fe8fa2 object_getIvar + 0 [chris-joness-mac-pro:00350] [ 8] 10 CoreFoundation 0x00007fff8143e631 -[NSObject(NSObject) finalize] + 129 [chris-joness-mac-pro:00350] [ 9] 11 mca_pls_xgrid.so 0x00000001002a9ce3 -[PlsXGridClient dealloc] + 419 [chris-joness-mac-pro:00350] [10] 12 mca_pls_xgrid.so 0x00000001002a9837 orte_pls_xgrid_finalize + 40 [chris-joness-mac-pro:00350] [11] 13 libopen-rte.0.dylib 0x000000010002d0f9 orte_pls_base_close + 249 [chris-joness-mac-pro:00350] [12] 14 libopen-rte.0.dylib 0x0000000100012027 orte_system_finalize + 119 [chris-joness-mac-pro:00350] [13] 15 libopen-rte.0.dylib 0x000000010000e968 orte_finalize + 40 [chris-joness-mac-pro:00350] [14] 16 mpirun 0x00000001000011ff orterun + 2042 [chris-joness-mac-pro:00350] [15] 17 mpirun 0x0000000100000a03 main + 27 [chris-joness-mac-pro:00350] [16] 18 mpirun 0x00000001000009e0 start + 52 [chris-joness-mac-pro:00350] [17] 19 ??? 0x0000000000000004 0x0 + 4 [chris-joness-mac-pro:00350] *** End of error message *** Abort trap I've seen this error in a previous mailing, and it seems that the issue has something to do with forcing everything to use kerberos (SSO). However, I noticed that in the computer being used as an agent, this option is grayed on in the Xgrid sharing configuration (I have no idea why). I would therefore ask if it is absolutely necessary to use SSO to get openmpi to run with xgrid, or am I missing something with the password setup. Seems that the kerberos option is much more complicated, and I may even want to switch to just using openmpi with ssh. Many thanks, Chris Chris Jones Post-doctoral Research Assistant, Department of Microbiology Swedish University of Agricultural Sciences Uppsala, Sweden phone: +46 (0)18 67 3222 email: chris.jo...@slu.se Department of Soil and Environmental Microbiology National Institute for Agronomic Research Dijon, France _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users-- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ ------------------------------ Message: 5 Date: Fri, 5 Aug 2011 08:41:58 -0500 From: Rob Latham<r...@mcs.anl.gov> Subject: Re: [OMPI users] parallel I/O on 64-bit indexed arays To: Open MPI Users<us...@open-mpi.org> Cc: Quincey Koziol<koz...@hdfgroup.org>, Fab Tillier <ftill...@microsoft.com> Message-ID:<20110805134158.ga28...@mcs.anl.gov> Content-Type: text/plain; charset=us-ascii On Wed, Jul 27, 2011 at 06:13:05PM +0200, Troels Haugboelle wrote:and we get good (+GB/s) performance when writing files from large runs. Interestingly, an alternative and conceptually simpler option is to use MPI_FILE_WRITE_ORDERED, but the performance of that function on Blue-Gene/P sucks - 20 MB/s instead of GB/s. I do not know why.Ordered mode as implemented in ROMIO is awful. Entirely serialized. We pass a token from process to process. Each process acquires the token, updates the shared file pointer, does its i/o, then passes the token to the next process. What we should do, and have done in test branches [1], is use MPI_SCAN to look at the shared file pointer once, tell all the processors their offset, then update the shared file pointer while all processes do I/O in parallel. [1]: Robert Latham, Robert Ross, and Rajeev Thakur. "Implementing MPI-IO Atomic Mode and Shared File Pointers Using MPI One-Sided Communication". International Journal of High Performance Computing Applications, 21(2):132-143, 2007 Since no one uses the shared file pointers, and even fewer people use ordered mode, we just haven't seen the need to do so. Do you want to rebuild your MPI library on BlueGene? I can pretty quickly generate and send a patch that will make ordered mode go whip fast. ==robTroels On 6/7/11 15:04 , Jeff Squyres wrote:On Jun 7, 2011, at 4:53 AM, Troels Haugboelle wrote:In principle yes, but the problem is we have an unequal amount of particles on each node, so the length of each array is not guaranteed to be divisible by 2, 4 or any other number. If I have understood the definition of MPI_TYPE_CREATE_SUBARRAY correctly the offset can be 64-bit, but not the global array size, so, optimally, what I am looking for is something that has unequal size for each thread, simple vector, and with 64-bit offsets and global array size.It's a bit awkward, but you can still make datatypes to give the offset that you want. E.g., if you need an offset of 2B+31 bytes, you can make datatype A with type contig of N=(2B/sizeof(int)) int's. Then make datatype B with type struct, containing type A and 31 MPI_BYTEs. Then use 1 instance of datatype B to get the offset that you want. You could make utility functions that, given a specific (64 bit) offset, it makes an MPI datatype that matches the offset, and then frees it (and all sub-datatypes). There is a bit of overhead in creating these datatypes, but it should be dwarfed by the amount of data that you're reading/writing, right? It's awkward, but it should work.Another possible workaround would be to identify subsections that do not pass 2B elements, make sub communicators, and then let each of them dump their elements with proper offsets. It may work. The problematic architecture is a BG/P. On other clusters doing simple I/O, letting all threads open the file, seek to their position, and then write their chunk works fine, but somehow on BG/P performance drops dramatically. My guess is that there is some file locking, or we are overwhelming the I/O nodes..This ticket for the MPI-3 standard is a first step in the right direction, but won't do everything you need (this is more FYI): https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/265 See the PDF attached to the ticket; it's going up for a "first reading" in a month. It'll hopefully be part of the MPI-3 standard by the end of the year (Fab Tillier, CC'ed, has been the chief proponent of this ticket for the past several months). Quincey Koziol from the HDF group is going to propose a follow on to this ticket, specifically about the case you're referring to -- large counts for file functions and datatype constructors. Quincey -- can you expand on what you'll be proposing, perchance?Interesting, I think something along the lines of the note would be very useful and needed for large applications. Thanks a lot for the pointers and your suggestions, cheers, Troels_______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA ------------------------------ _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users End of users Digest, Vol 1977, Issue 1 **************************************_______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
<<attachment: warren.vcf>>
