Have you setup your shell startup files such that they point to the new OMPI installation (/opt/local/openmpi/) even for non-interactive logins?
On Aug 10, 2011, at 6:14 AM, Christopher Jones wrote: > Hi, > > Thanks for the quick response.....I managed to compile 1.5.3 on both > computers using gcc-4.2, with the proper flags set (this took a bit of > playing with, but I did eventually get it to compile). Once that was done, I > installed it to a different directory from 1.2.8 (/opt/local/openmpi/), > specified the PATH and LD_LIBRARY_PATH for the new version on each computer, > then managed to get the hello_world script to run again so it could call each > process, like before. However, I'm still in the same place - ring_c freezes > up. I tried changing the hostname in the host file (just for poops and > giggles - I see the response stating it doesn't matter), but to no avail. I > made sure the firewall is off on both computers. > > I'm hoping I'm not doing something overly dumb here, but I'm still a bit > stuck...I see in the FAQ that there were some issues with nehalem processors > - I have two Xeons in one box and a nehalem in another. Could this make any > difference? > > Thanks again, > Chris > > On Aug 9, 2011, at 6:50 PM, Jeff Squyres wrote: > >> No, Open MPI doesn't use the names in the hostfile to figure out which >> TCP/IP addresses to use (for example). Each process ends up publishing a >> list of IP addresses at which it can be connected, and OMPI does routability >> computations to figure out which is the "best" address to contact a given >> peer on. >> >> If you're just starting with Open MPI, can you upgrade? 1.2.8 is pretty >> ancient. Open MPI 1.4.3 is the most recent stable release; 1.5.3 is our >> "feature" series, but it's also relatively stable (new releases are coming >> in both the 1.4.x and 1.5.x series soon, FWIW). >> >> >> On Aug 9, 2011, at 12:14 PM, David Warren wrote: >> >>> I don't know if this is it, but if you use the name localhost, won't >>> processes on both machines try to talk to 127.0.0.1? I believe you need to >>> use the real hostname in you host file. I think that your two tests work >>> because there is no interprocess communication, just stdout. >>> >>> On 08/08/11 23:46, Christopher Jones wrote: >>>> Hi again, >>>> >>>> I changed the subject of my previous posting to reflect a new problem >>>> encountered when I changed my strategy to using SSH instead of Xgrid on >>>> two mac pros. I've set up a login-less ssh communication between the two >>>> macs (connected via direct ethernet, both running openmpi 1.2.8 on OSX >>>> 10.6.8) per the instructions on the FAQ. I can type in 'ssh >>>> computer-name.local' on either computer and connect without a password >>>> prompt. From what I can see, the ssh-agent is up and running - the >>>> following is listed in my ENV: >>>> >>>> SSH_AUTH_SOCK=/tmp/launch-5FoCc1/Listeners >>>> SSH_AGENT_PID=61058 >>>> >>>> My host file simply lists 'localhost' and >>>> 'chrisjones2@allana-welshs-mac-pro.local'. When I run a simple hello_world >>>> test, I get what seems like a reasonable output: >>>> >>>> chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile >>>> ./test_hello >>>> Hello world from process 0 of 8 >>>> Hello world from process 1 of 8 >>>> Hello world from process 2 of 8 >>>> Hello world from process 3 of 8 >>>> Hello world from process 4 of 8 >>>> Hello world from process 7 of 8 >>>> Hello world from process 5 of 8 >>>> Hello world from process 6 of 8 >>>> >>>> I can also run hostname and get what seems to be an ok response (unless >>>> I'm wrong about this): >>>> >>>> chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile hostname >>>> allana-welshs-mac-pro.local >>>> allana-welshs-mac-pro.local >>>> allana-welshs-mac-pro.local >>>> allana-welshs-mac-pro.local >>>> quadcore.mikrob.slu.se >>>> quadcore.mikrob.slu.se >>>> quadcore.mikrob.slu.se >>>> quadcore.mikrob.slu.se >>>> >>>> >>>> However, when I run the ring_c test, it freezes: >>>> >>>> chris-joness-mac-pro:~ chrisjones$ mpirun -np 8 -hostfile hostfile ./ring_c >>>> Process 0 sending 10 to 1, tag 201 (8 processes in ring) >>>> Process 0 sent to 1 >>>> Process 0 decremented value: 9 >>>> >>>> (I noted that processors on both computers are active). >>>> >>>> ring_c was compiled separately on each computer, however both have the >>>> same version of openmpi and OSX. I've gone through the FAQ and searched >>>> the user forum, but I can't quite seems to get this problem unstuck. >>>> >>>> Many thanks for your time, >>>> Chris >>>> >>>> On Aug 5, 2011, at 6:00 PM,<users-requ...@open-mpi.org> >>>> <users-requ...@open-mpi.org> wrote: >>>> >>>> >>>>> Send users mailing list submissions to >>>>> us...@open-mpi.org >>>>> >>>>> To subscribe or unsubscribe via the World Wide Web, visit >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> or, via email, send a message with subject or body 'help' to >>>>> users-requ...@open-mpi.org >>>>> >>>>> You can reach the person managing the list at >>>>> users-ow...@open-mpi.org >>>>> >>>>> When replying, please edit your Subject line so it is more specific >>>>> than "Re: Contents of users digest..." >>>>> >>>>> >>>>> Today's Topics: >>>>> >>>>> 1. Re: OpenMPI causing WRF to crash (Jeff Squyres) >>>>> 2. Re: OpenMPI causing WRF to crash (Anthony Chan) >>>>> 3. Re: Program hangs on send when run with nodes on remote >>>>> machine (Jeff Squyres) >>>>> 4. Re: openmpi 1.2.8 on Xgrid noob issue (Jeff Squyres) >>>>> 5. Re: parallel I/O on 64-bit indexed arays (Rob Latham) >>>>> >>>>> >>>>> ---------------------------------------------------------------------- >>>>> >>>>> Message: 1 >>>>> Date: Thu, 4 Aug 2011 19:18:36 -0400 >>>>> From: Jeff Squyres<jsquy...@cisco.com> >>>>> Subject: Re: [OMPI users] OpenMPI causing WRF to crash >>>>> To: Open MPI Users<us...@open-mpi.org> >>>>> Message-ID:<3f0e661f-a74f-4e51-86c0-1f84feb07...@cisco.com> >>>>> Content-Type: text/plain; charset=windows-1252 >>>>> >>>>> Signal 15 is usually SIGTERM on Linux, meaning that some external entity >>>>> probably killed the job. >>>>> >>>>> The OMPI error message you describe is also typical for that kind of >>>>> scenario -- i.e., a process exited without calling MPI_Finalize could >>>>> mean that it called exit() or some external process killed it. >>>>> >>>>> >>>>> On Aug 3, 2011, at 7:24 AM, BasitAli Khan wrote: >>>>> >>>>> >>>>>> I am trying to run a rather heavy wrf simulation with spectral nudging >>>>>> but the simulation crashes after 1.8 minutes of integration. >>>>>> The simulation has two domains with d01 = 601x601 and d02 = 721x721 >>>>>> and 51 vertical levels. I tried this simulation on two different systems >>>>>> but result was more or less same. For example >>>>>> >>>>>> On our Bluegene/P with SUSE Linux Enterprise Server 10 ppc and XLF >>>>>> compiler I tried to run wrf on 2048 shared memory nodes (1 compute node >>>>>> = 4 cores , 32 bit, 850 Mhz). For the parallel run I used mpixlc, >>>>>> mpixlcxx and mpixlf90. I got the following error message in the wrf.err >>>>>> file >>>>>> >>>>>> <Aug 01 19:50:21.244540> BE_MPI (ERROR): The error message in the job >>>>>> record is as follows: >>>>>> <Aug 01 19:50:21.244657> BE_MPI (ERROR): "killed with signal 15" >>>>>> >>>>>> I also tried to run the same simulation on our linux cluster (Linux Red >>>>>> Hat Enterprise 5.4m x86_64 and Intel compiler) with 8, 16 and 64 nodes >>>>>> (1 compute node=8 cores). For the parallel run I am used >>>>>> mpi/openmpi/1.4.2-intel-11. I got the following error message in the >>>>>> error log after couple of minutes of integration. >>>>>> >>>>>> "mpirun has exited due to process rank 45 with PID 19540 on >>>>>> node ci118 exiting without calling "finalize". This may >>>>>> have caused other processes in the application to be >>>>>> terminated by signals sent by mpirun (as reported here)." >>>>>> >>>>>> I tried many things but nothing seems to be working. However, if I >>>>>> reduce grid points below 200, the simulation goes fine. It appears that >>>>>> probably OpenMP has problem with large number of grid points but I have >>>>>> no idea how to fix it. I will greatly appreciate if you could suggest >>>>>> some solution. >>>>>> >>>>>> Best regards, >>>>>> --- >>>>>> Basit A. Khan, Ph.D. >>>>>> Postdoctoral Fellow >>>>>> Division of Physical Sciences& Engineering >>>>>> Office# 3204, Level 3, Building 1, >>>>>> King Abdullah University of Science& Technology >>>>>> 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, >>>>>> Kingdom of Saudi Arabia. >>>>>> >>>>>> Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 >>>>>> E-mail: basitali.k...@kaust.edu.sa >>>>>> Skype name: basit.a.khan >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>>> -- >>>>> Jeff Squyres >>>>> jsquy...@cisco.com >>>>> For corporate legal information go to: >>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> Message: 2 >>>>> Date: Thu, 4 Aug 2011 18:59:59 -0500 (CDT) >>>>> From: Anthony Chan<c...@mcs.anl.gov> >>>>> Subject: Re: [OMPI users] OpenMPI causing WRF to crash >>>>> To: Open MPI Users<us...@open-mpi.org> >>>>> Message-ID: >>>>> <660521091.191111.1312502399225.javamail.r...@zimbra.anl.gov> >>>>> Content-Type: text/plain; charset=utf-8 >>>>> >>>>> >>>>> If you want to debug this on BGP, you could set BG_COREDUMPONERROR=1 >>>>> and look at the backtrace in the light weight core files >>>>> (you probably need to recompile everything with -g). >>>>> >>>>> A.Chan >>>>> >>>>> ----- Original Message ----- >>>>> >>>>>> Hi Dmitry, >>>>>> Thanks for a prompt and fairly detailed response. I have also >>>>>> forwarded >>>>>> the email to wrf community in the hope that somebody would have some >>>>>> straight forward solution. I will try to debug the error as suggested >>>>>> by >>>>>> you if I would not have much luck from the wrf forum. >>>>>> >>>>>> Cheers, >>>>>> --- >>>>>> >>>>>> Basit A. Khan, Ph.D. >>>>>> Postdoctoral Fellow >>>>>> Division of Physical Sciences& Engineering >>>>>> Office# 3204, Level 3, Building 1, >>>>>> King Abdullah University of Science& Technology >>>>>> 4700 King Abdullah Blvd, Box 2753, Thuwal 23955 ?6900, >>>>>> Kingdom of Saudi Arabia. >>>>>> >>>>>> Office: +966(0)2 808 0276, Mobile: +966(0)5 9538 7592 >>>>>> E-mail: basitali.k...@kaust.edu.sa >>>>>> Skype name: basit.a.khan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 8/3/11 2:46 PM, "Dmitry N. Mikushin"<maemar...@gmail.com> wrote: >>>>>> >>>>>> >>>>>>> 5 apparently means one of the WRF's MPI processes has been >>>>>>> unexpectedly terminated, maybe by program decision. No matter, if it >>>>>>> is OpenMPI-specifi >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> Message: 3 >>>>> Date: Thu, 4 Aug 2011 20:46:16 -0400 >>>>> From: Jeff Squyres<jsquy...@cisco.com> >>>>> Subject: Re: [OMPI users] Program hangs on send when run with nodes on >>>>> remote machine >>>>> To: Open MPI Users<us...@open-mpi.org> >>>>> Message-ID:<f344f301-ad7b-4e83-b0df-a6e001072...@cisco.com> >>>>> Content-Type: text/plain; charset=us-ascii >>>>> >>>>> I notice that in the worker, you have: >>>>> >>>>> eth2 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d4 >>>>> inet addr:192.168.1.155 Bcast:192.168.1.255 Mask:255.255.255.0 >>>>> inet6 addr: fe80::21b:21ff:fe77:c5d4/64 Scope:Link >>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>>>> RX packets:9225846 errors:0 dropped:75175 overruns:0 frame:0 >>>>> TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 >>>>> collisions:0 txqueuelen:1000 >>>>> RX bytes:1336628768 (1.3 GB) TX bytes:552 (552.0 B) >>>>> >>>>> eth3 Link encap:Ethernet HWaddr 00:1b:21:77:c5:d5 >>>>> inet addr:192.168.1.156 Bcast:192.168.1.255 Mask:255.255.255.0 >>>>> inet6 addr: fe80::21b:21ff:fe77:c5d5/64 Scope:Link >>>>> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >>>>> RX packets:26481809 errors:0 dropped:75059 overruns:0 frame:0 >>>>> TX packets:18030236 errors:0 dropped:0 overruns:0 carrier:0 >>>>> collisions:0 txqueuelen:1000 >>>>> RX bytes:70061260271 (70.0 GB) TX bytes:11844181778 (11.8 GB) >>>>> >>>>> Two different NICs are on the same subnet -- that doesn't seem like a >>>>> good idea...? I think this topic has come up on the users list before, >>>>> and, IIRC, the general consensus is "don't do that" because it's not >>>>> clear as to which NIC Linux will actually send outgoing traffic across >>>>> bound for the 192.168.1.x subnet. >>>>> >>>>> >>>>> >>>>> On Aug 4, 2011, at 1:59 PM, Keith Manville wrote: >>>>> >>>>> >>>>>> I am having trouble running my MPI program on multiple nodes. I can >>>>>> run multiple processes on a single node, and I can spawn processes on >>>>>> on remote nodes, but when I call Send from a remote node, the node >>>>>> never returns, even though there is an appropriate Recv waiting. I'm >>>>>> pretty sure this is an issue with my configuration, not my code. I've >>>>>> tried some other sample programs I found and had the same problem of >>>>>> hanging on a send from one host to another. >>>>>> >>>>>> Here's an in depth description: >>>>>> >>>>>> I wrote a quick test program where each process with rank> 1 sends an >>>>>> int to the master (rank 0), and the master receives until it gets >>>>>> something from every other process. >>>>>> >>>>>> My test program works fine when I run multiple processes on a single >>>>>> machine. >>>>>> >>>>>> either the local node: >>>>>> >>>>>> $ ./mpirun -n 4 ./mpi-test >>>>>> Hi I'm localhost:2 >>>>>> Hi I'm localhost:1 >>>>>> localhost:1 sending 11... >>>>>> localhost:2 sending 12... >>>>>> localhost:2 sent 12 >>>>>> localhost:1 sent 11 >>>>>> Hi I'm localhost:0 >>>>>> localhost:0 received 11 from 1 >>>>>> localhost:0 received 12 from 2 >>>>>> Hi I'm localhost:3 >>>>>> localhost:3 sending 13... >>>>>> localhost:3 sent 13 >>>>>> localhost:0 received 13 from 3 >>>>>> all workers checked in! >>>>>> >>>>>> or a remote one: >>>>>> >>>>>> $ ./mpirun -np 2 -host remotehost ./mpi-test >>>>>> Hi I'm remotehost:0 >>>>>> remotehost:0 received 11 from 1 >>>>>> all workers checked in! >>>>>> Hi I'm remotehost:1 >>>>>> remotehost:1 sending 11... >>>>>> remotehost:1 sent 11 >>>>>> >>>>>> But when I try to run the master locally and the worker(s) remotely >>>>>> (this is the way I am actually interested in running it), Send never >>>>>> returns and it hangs indefinitely. >>>>>> >>>>>> $ ./mpirun -np 2 -host localhost,remotehost ./mpi-test >>>>>> Hi I'm localhost:0 >>>>>> Hi I'm remotehost:1 >>>>>> remotehost:1 sending 11... >>>>>> >>>>>> Just to see if it would work, I tried spawning the master on the >>>>>> remotehost and the worker on the localhost. >>>>>> >>>>>> $ ./mpirun -np 2 -host remotehost,localhost ./mpi-test >>>>>> Hi I'm localhost:1 >>>>>> localhost:1 sending 11... >>>>>> localhost:1 sent 11 >>>>>> Hi I'm remotehost:0 >>>>>> remotehost:0 received 0 from 1 >>>>>> all workers checked in! >>>>>> >>>>>> It doesn't hang on Send, but the wrong value is received. >>>>>> >>>>>> Any idea what's going on? I've attached my code, my config.log, >>>>>> ifconfig output, and ompi_info output. >>>>>> >>>>>> Thanks, >>>>>> Keith >>>>>> <mpi.tgz>_______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>>> -- >>>>> Jeff Squyres >>>>> jsquy...@cisco.com >>>>> For corporate legal information go to: >>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> Message: 4 >>>>> Date: Thu, 4 Aug 2011 20:48:30 -0400 >>>>> From: Jeff Squyres<jsquy...@cisco.com> >>>>> Subject: Re: [OMPI users] openmpi 1.2.8 on Xgrid noob issue >>>>> To: Open MPI Users<us...@open-mpi.org> >>>>> Message-ID:<c2ea7fd0-badb-4d05-851c-c444be26f...@cisco.com> >>>>> Content-Type: text/plain; charset=us-ascii >>>>> >>>>> I'm afraid our Xgrid support has lagged, and Apple hasn't show much >>>>> interest in MPI + Xgrid support -- much less HPC. :-\ >>>>> >>>>> Have you see the FAQ items about Xgrid? >>>>> >>>>> http://www.open-mpi.org/faq/?category=osx#xgrid-howto >>>>> >>>>> >>>>> On Aug 4, 2011, at 4:16 AM, Christopher Jones wrote: >>>>> >>>>> >>>>>> Hi there, >>>>>> >>>>>> I'm currently trying to set up a small xgrid between two mac pros (a >>>>>> single quadcore and a 2 duo core), where both are directly connected via >>>>>> an ethernet cable. I've set up xgrid using the password authentication >>>>>> (rather than the kerberos), and from what I can tell in the Xgrid admin >>>>>> tool it seems to be working. However, once I try a simple hello world >>>>>> program, I get this error: >>>>>> >>>>>> chris-joness-mac-pro:~ chrisjones$ mpirun -np 4 ./test_hello >>>>>> mpirun noticed that job rank 0 with PID 381 on node xgrid-node-0 exited >>>>>> on signal 15 (Terminated). >>>>>> 1 additional process aborted (not shown) >>>>>> 2011-08-04 10:02:16.329 mpirun[350:903] *** Terminating app due to >>>>>> uncaught exception 'NSInvalidArgumentException', reason: '*** >>>>>> -[NSKVONotifying_XGConnection<0x1001325a0> finalize]: called when >>>>>> collecting not enabled' >>>>>> *** Call stack at first throw: >>>>>> ( >>>>>> 0 CoreFoundation 0x00007fff814237b4 >>>>>> __exceptionPreprocess + 180 >>>>>> 1 libobjc.A.dylib 0x00007fff84fe8f03 >>>>>> objc_exception_throw + 45 >>>>>> 2 CoreFoundation 0x00007fff8143e631 >>>>>> -[NSObject(NSObject) finalize] + 129 >>>>>> 3 mca_pls_xgrid.so 0x00000001002a9ce3 >>>>>> -[PlsXGridClient dealloc] + 419 >>>>>> 4 mca_pls_xgrid.so 0x00000001002a9837 >>>>>> orte_pls_xgrid_finalize + 40 >>>>>> 5 libopen-rte.0.dylib 0x000000010002d0f9 >>>>>> orte_pls_base_close + 249 >>>>>> 6 libopen-rte.0.dylib 0x0000000100012027 >>>>>> orte_system_finalize + 119 >>>>>> 7 libopen-rte.0.dylib 0x000000010000e968 >>>>>> orte_finalize + 40 >>>>>> 8 mpirun 0x00000001000011ff orterun + >>>>>> 2042 >>>>>> 9 mpirun 0x0000000100000a03 main + 27 >>>>>> 10 mpirun 0x00000001000009e0 start + 52 >>>>>> 11 ??? 0x0000000000000004 0x0 + 4 >>>>>> ) >>>>>> terminate called after throwing an instance of 'NSException' >>>>>> [chris-joness-mac-pro:00350] *** Process received signal *** >>>>>> [chris-joness-mac-pro:00350] Signal: Abort trap (6) >>>>>> [chris-joness-mac-pro:00350] Signal code: (0) >>>>>> [chris-joness-mac-pro:00350] [ 0] 2 libSystem.B.dylib >>>>>> 0x00007fff81ca51ba _sigtramp + 26 >>>>>> [chris-joness-mac-pro:00350] [ 1] 3 ??? >>>>>> 0x00000001000cd400 0x0 + 4295808000 >>>>>> [chris-joness-mac-pro:00350] [ 2] 4 libstdc++.6.dylib >>>>>> 0x00007fff830965d2 __tcf_0 + 0 >>>>>> [chris-joness-mac-pro:00350] [ 3] 5 libobjc.A.dylib >>>>>> 0x00007fff84fecb39 _objc_terminate + 100 >>>>>> [chris-joness-mac-pro:00350] [ 4] 6 libstdc++.6.dylib >>>>>> 0x00007fff83094ae1 _ZN10__cxxabiv111__terminateEPFvvE + 11 >>>>>> [chris-joness-mac-pro:00350] [ 5] 7 libstdc++.6.dylib >>>>>> 0x00007fff83094b16 _ZN10__cxxabiv112__unexpectedEPFvvE + 0 >>>>>> [chris-joness-mac-pro:00350] [ 6] 8 libstdc++.6.dylib >>>>>> 0x00007fff83094bfc >>>>>> _ZL23__gxx_exception_cleanup19_Unwind_Reason_CodeP17_Unwind_Exception + 0 >>>>>> [chris-joness-mac-pro:00350] [ 7] 9 libobjc.A.dylib >>>>>> 0x00007fff84fe8fa2 object_getIvar + 0 >>>>>> [chris-joness-mac-pro:00350] [ 8] 10 CoreFoundation >>>>>> 0x00007fff8143e631 -[NSObject(NSObject) finalize] + 129 >>>>>> [chris-joness-mac-pro:00350] [ 9] 11 mca_pls_xgrid.so >>>>>> 0x00000001002a9ce3 -[PlsXGridClient dealloc] + 419 >>>>>> [chris-joness-mac-pro:00350] [10] 12 mca_pls_xgrid.so >>>>>> 0x00000001002a9837 orte_pls_xgrid_finalize + 40 >>>>>> [chris-joness-mac-pro:00350] [11] 13 libopen-rte.0.dylib >>>>>> 0x000000010002d0f9 orte_pls_base_close + 249 >>>>>> [chris-joness-mac-pro:00350] [12] 14 libopen-rte.0.dylib >>>>>> 0x0000000100012027 orte_system_finalize + 119 >>>>>> [chris-joness-mac-pro:00350] [13] 15 libopen-rte.0.dylib >>>>>> 0x000000010000e968 orte_finalize + 40 >>>>>> [chris-joness-mac-pro:00350] [14] 16 mpirun >>>>>> 0x00000001000011ff orterun + 2042 >>>>>> [chris-joness-mac-pro:00350] [15] 17 mpirun >>>>>> 0x0000000100000a03 main + 27 >>>>>> [chris-joness-mac-pro:00350] [16] 18 mpirun >>>>>> 0x00000001000009e0 start + 52 >>>>>> [chris-joness-mac-pro:00350] [17] 19 ??? >>>>>> 0x0000000000000004 0x0 + 4 >>>>>> [chris-joness-mac-pro:00350] *** End of error message *** >>>>>> Abort trap >>>>>> >>>>>> >>>>>> I've seen this error in a previous mailing, and it seems that the issue >>>>>> has something to do with forcing everything to use kerberos (SSO). >>>>>> However, I noticed that in the computer being used as an agent, this >>>>>> option is grayed on in the Xgrid sharing configuration (I have no idea >>>>>> why). I would therefore ask if it is absolutely necessary to use SSO to >>>>>> get openmpi to run with xgrid, or am I missing something with the >>>>>> password setup. Seems that the kerberos option is much more complicated, >>>>>> and I may even want to switch to just using openmpi with ssh. >>>>>> >>>>>> Many thanks, >>>>>> Chris >>>>>> >>>>>> >>>>>> Chris Jones >>>>>> Post-doctoral Research Assistant, >>>>>> >>>>>> Department of Microbiology >>>>>> Swedish University of Agricultural Sciences >>>>>> Uppsala, Sweden >>>>>> phone: +46 (0)18 67 3222 >>>>>> email: chris.jo...@slu.se >>>>>> >>>>>> Department of Soil and Environmental Microbiology >>>>>> National Institute for Agronomic Research >>>>>> Dijon, France >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> >>>>> -- >>>>> Jeff Squyres >>>>> jsquy...@cisco.com >>>>> For corporate legal information go to: >>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> Message: 5 >>>>> Date: Fri, 5 Aug 2011 08:41:58 -0500 >>>>> From: Rob Latham<r...@mcs.anl.gov> >>>>> Subject: Re: [OMPI users] parallel I/O on 64-bit indexed arays >>>>> To: Open MPI Users<us...@open-mpi.org> >>>>> Cc: Quincey Koziol<koz...@hdfgroup.org>, Fab Tillier >>>>> <ftill...@microsoft.com> >>>>> Message-ID:<20110805134158.ga28...@mcs.anl.gov> >>>>> Content-Type: text/plain; charset=us-ascii >>>>> >>>>> On Wed, Jul 27, 2011 at 06:13:05PM +0200, Troels Haugboelle wrote: >>>>> >>>>>> and we get good (+GB/s) performance when writing files from large runs. >>>>>> >>>>>> Interestingly, an alternative and conceptually simpler option is to >>>>>> use MPI_FILE_WRITE_ORDERED, but the performance of that function on >>>>>> Blue-Gene/P sucks - 20 MB/s instead of GB/s. I do not know why. >>>>>> >>>>> Ordered mode as implemented in ROMIO is awful. Entirely serialized. >>>>> We pass a token from process to process. Each process acquires the >>>>> token, updates the shared file pointer, does its i/o, then passes the >>>>> token to the next process. >>>>> >>>>> What we should do, and have done in test branches [1], is use MPI_SCAN >>>>> to look at the shared file pointer once, tell all the processors their >>>>> offset, then update the shared file pointer while all processes do I/O >>>>> in parallel. >>>>> >>>>> [1]: Robert Latham, Robert Ross, and Rajeev Thakur. "Implementing >>>>> MPI-IO Atomic Mode and Shared File Pointers Using MPI One-Sided >>>>> Communication". International Journal of High Performance Computing >>>>> Applications, 21(2):132-143, 2007 >>>>> >>>>> Since no one uses the shared file pointers, and even fewer people use >>>>> ordered mode, we just haven't seen the need to do so. >>>>> >>>>> Do you want to rebuild your MPI library on BlueGene? I can pretty >>>>> quickly generate and send a patch that will make ordered mode go whip >>>>> fast. >>>>> >>>>> ==rob >>>>> >>>>> >>>>>> Troels >>>>>> >>>>>> On 6/7/11 15:04 , Jeff Squyres wrote: >>>>>> >>>>>>> On Jun 7, 2011, at 4:53 AM, Troels Haugboelle wrote: >>>>>>> >>>>>>> >>>>>>>> In principle yes, but the problem is we have an unequal amount of >>>>>>>> particles on each node, so the length of each array is not guaranteed >>>>>>>> to be divisible by 2, 4 or any other number. If I have understood the >>>>>>>> definition of MPI_TYPE_CREATE_SUBARRAY correctly the offset can be >>>>>>>> 64-bit, but not the global array size, so, optimally, what I am >>>>>>>> looking for is something that has unequal size for each thread, simple >>>>>>>> vector, and with 64-bit offsets and global array size. >>>>>>>> >>>>>>> It's a bit awkward, but you can still make datatypes to give the offset >>>>>>> that you want. E.g., if you need an offset of 2B+31 bytes, you can >>>>>>> make datatype A with type contig of N=(2B/sizeof(int)) int's. Then >>>>>>> make datatype B with type struct, containing type A and 31 MPI_BYTEs. >>>>>>> Then use 1 instance of datatype B to get the offset that you want. >>>>>>> >>>>>>> You could make utility functions that, given a specific (64 bit) >>>>>>> offset, it makes an MPI datatype that matches the offset, and then >>>>>>> frees it (and all sub-datatypes). >>>>>>> >>>>>>> There is a bit of overhead in creating these datatypes, but it should >>>>>>> be dwarfed by the amount of data that you're reading/writing, right? >>>>>>> >>>>>>> It's awkward, but it should work. >>>>>>> >>>>>>> >>>>>>>> Another possible workaround would be to identify subsections that do >>>>>>>> not pass 2B elements, make sub communicators, and then let each of >>>>>>>> them dump their elements with proper offsets. It may work. The >>>>>>>> problematic architecture is a BG/P. On other clusters doing simple >>>>>>>> I/O, letting all threads open the file, seek to their position, and >>>>>>>> then write their chunk works fine, but somehow on BG/P performance >>>>>>>> drops dramatically. My guess is that there is some file locking, or we >>>>>>>> are overwhelming the I/O nodes.. >>>>>>>> >>>>>>>> >>>>>>>>> This ticket for the MPI-3 standard is a first step in the right >>>>>>>>> direction, but won't do everything you need (this is more FYI): >>>>>>>>> >>>>>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/265 >>>>>>>>> >>>>>>>>> See the PDF attached to the ticket; it's going up for a "first >>>>>>>>> reading" in a month. It'll hopefully be part of the MPI-3 standard >>>>>>>>> by the end of the year (Fab Tillier, CC'ed, has been the chief >>>>>>>>> proponent of this ticket for the past several months). >>>>>>>>> >>>>>>>>> Quincey Koziol from the HDF group is going to propose a follow on to >>>>>>>>> this ticket, specifically about the case you're referring to -- large >>>>>>>>> counts for file functions and datatype constructors. Quincey -- can >>>>>>>>> you expand on what you'll be proposing, perchance? >>>>>>>>> >>>>>>>> Interesting, I think something along the lines of the note would be >>>>>>>> very useful and needed for large applications. >>>>>>>> >>>>>>>> Thanks a lot for the pointers and your suggestions, >>>>>>>> >>>>>>>> cheers, >>>>>>>> >>>>>>>> Troels >>>>>>>> >>>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>> -- >>>>> Rob Latham >>>>> Mathematics and Computer Science Division >>>>> Argonne National Lab, IL USA >>>>> >>>>> >>>>> ------------------------------ >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> End of users Digest, Vol 1977, Issue 1 >>>>> ************************************** >>>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> <warren.vcf>_______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
