Re: [OMPI users] INSTALL bug in 64-bit build of OpenMPI Release build on Windows - has workaround
Shiqing, Damien, > If you already have an x86 solution, and you want to have > another for x64, you have to start over from the CMake-GUI, select the > 64 bit generator, i.e. "Visual Studio 9 2008 64bit", so that to generate > the a new solution in a different directory. That was the source of my problem. I had overlooked the other Visual Studio 9 2008 option, thinking there was just one for the package -- and ended up thinking I had to switch using the configuration manager in VS2008 itself. Thanks! Marcus
Re: [OMPI users] Anybody built a working 1.4.1 on Solaris 8 (Sparc)?
On Feb 4, 2010, at 4:52 PM, David Mathog wrote: Has anybody built 1.4.1 on Solaris 8 (Sparc), because it isn't going very well here. If you succeeded at this please tell me how you did it. Here is my tale of woe. First attempt with gcc (3.4.6 from SunFreeware) and ./configure --with-sge --prefix=/opt/ompi141 I think you need to add --enable-heterogeneous because you are trying to run this on a cluster of SPARC and IA32 machines. (You will also need to use that configure flag to build OMPI on your IA32 machines.) Iain
Re: [OMPI users] Anybody built a working 1.4.1 on Solaris 8 (Sparc)?
We haven't tried Solaris 8 in quite some time. However, for your first issue did you include the --enable-heterogeneous option on your configure command? Since you are mix IA-32 and SPARC nodes you'll want to include this so the endian issue doesn't bite you. --td Message: 5 Date: Thu, 04 Feb 2010 13:52:05 -0800 From: "David Mathog" Subject: [OMPI users] Anybody built a working 1.4.1 on Solaris 8 (Sparc)? To: us...@open-mpi.org Message-ID: Content-Type: text/plain; charset=iso-8859-1 Has anybody built 1.4.1 on Solaris 8 (Sparc), because it isn't going very well here. If you succeeded at this please tell me how you did it. Here is my tale of woe. First attempt with gcc (3.4.6 from SunFreeware) and ./configure --with-sge --prefix=/opt/ompi141 gmake all install >build_log1 2>&1 built ok (needed the same source changes described by Brian Blank for 1.3.1 or it wouldn't compile http://www.open-mpi.org/community/lists/users/2009/02/7994.php ), and mpirun worked OK with hello_c so long as it only sent jobs to the same machine like this: cat >justme&1 & gmake all install >build_log2 2>&1 and it had problems with vt_tracefilter.cc, where it stopped at: "The compiler currently does not permit non-POD variables ("name") in OpenMP regions". This was because of -DVT_OMP on the compilation line. Removed it, then the compiler did not include omp.h (ancient OpenMPI header file that came with Forte 7), compiled that one file manually, restarted the gmake and hit: /opt/SUNWspro/bin/CC -DHAVE_CONFIG_H -I. -I../.. \ -I../../extlib/otf/otflib -I../../extlib/otf/otflib \ -I../../vtlib/ -xopenmp -DVT_OMP -xarch=v8plusa -c \ -o vtunify-vt_unify.o \ `test -f 'vt_unify.cc' || echo './'`vt_unify.cc CC: Warning: Optimizer level changed from 0 to 3 to support parallelized code "vt_unify_stats.h", line 70: Error: m_iFuncStatSortFlags is not accessible from Statistics::FuncStat_struct::operator<(const Statistics::FuncStat_struct&) const. 1 Error(s) detected. This error didn't go away when -DVT_OMP was removed, nor when -xopenmp was also taken away, and so the final score is: working OpenMPI 1.4.1 on Solaris = 0 for 2 attempts. Thanks, David Mathog mat...@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech
[OMPI users] Infiniband Question
Howdy, When running a Gromacs job using OpenMPI 1.4.1 on Infiniband enabled nodes, I'm seeing the following process listing: \_ -bash /opt/gridengine/default/spool/compute-0-3/job_scripts/97037 \_ mpirun -np 4 mdrun_mpi -v -np 4 -s production-Npt-323K_4CPU -o production-Npt-323K_4CPU -c production-Npt-323K_4CPU -x production-Npt-323K_4CPU -g production-Npt-323K_4CPU.log \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V compute-0-4.local orted -mca ess env -mca orte_ess_jobid 945881088 -mca orte_ess_vpid 1 -mca orte_ess_num_procs 4 --hnp-uri "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V compute-0-2.local orted -mca ess env -mca orte_ess_jobid 945881088 -mca orte_ess_vpid 2 -mca orte_ess_num_procs 4 --hnp-uri "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V compute-0-1.local orted -mca ess env -mca orte_ess_jobid 945881088 -mca orte_ess_vpid 3 -mca orte_ess_num_procs 4 --hnp-uri "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" \_ mdrun_mpi -v -np 4 -s production-Npt-323K_4CPU -o production-Npt-323K_4CPU -c production-Npt-323K_4CPU -x production-Npt-323K_4CPU -g production-Npt-323K_4CPU.log Is it normal for these tcp addresses to be listed if the job is using Infiniband? The 192.168.20.x subnet is the eth0 GigE network And the 192.168.21.x subnet is the ib0 IPoverIB network Or is this job actually using TCPIP over Infiniband / GigE? I'm running mpirun without any special fabric includes / excludes. ompi_info lists openib as a valid fabric: $ ompi_info |grep openib MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.1) Thanks for any insight, Mike = Mike Hanby mha...@uab.edu Information Systems Specialist II IT HPCS / Research Computing
Re: [OMPI users] Infiniband Question
Yep -- it's normal. Those IP addresses are used for bootstrapping/startup, not for MPI traffic. In particular, that "HNP URI" stuff is used by Open MPI's underlying run-time environment. It's not used by the MPI layer at all. On Feb 5, 2010, at 2:32 PM, Mike Hanby wrote: > Howdy, > > When running a Gromacs job using OpenMPI 1.4.1 on Infiniband enabled nodes, > I'm seeing the following process listing: > > \_ -bash /opt/gridengine/default/spool/compute-0-3/job_scripts/97037 > \_ mpirun -np 4 mdrun_mpi -v -np 4 -s production-Npt-323K_4CPU -o > production-Npt-323K_4CPU -c production-Npt-323K_4CPU -x > production-Npt-323K_4CPU -g production-Npt-323K_4CPU.log > \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V > compute-0-4.local orted -mca ess env -mca orte_ess_jobid 945881088 > -mca orte_ess_vpid 1 -mca orte_ess_num_procs 4 --hnp-uri > "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" > \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V > compute-0-2.local orted -mca ess env -mca orte_ess_jobid 945881088 > -mca orte_ess_vpid 2 -mca orte_ess_num_procs 4 --hnp-uri > "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" > \_ /opt/gridengine/bin/lx26-amd64/qrsh -inherit -nostdin -V > compute-0-1.local orted -mca ess env -mca orte_ess_jobid 945881088 > -mca orte_ess_vpid 3 -mca orte_ess_num_procs 4 --hnp-uri > "945881088.0;tcp://192.168.20.252:39440;tcp://192.168.21.252:39440" > \_ mdrun_mpi -v -np 4 -s production-Npt-323K_4CPU -o > production-Npt-323K_4CPU -c production-Npt-323K_4CPU > -x production-Npt-323K_4CPU -g production-Npt-323K_4CPU.log > > Is it normal for these tcp addresses to be listed if the job is using > Infiniband? > > The 192.168.20.x subnet is the eth0 GigE network > And the 192.168.21.x subnet is the ib0 IPoverIB network > > Or is this job actually using TCPIP over Infiniband / GigE? > > I'm running mpirun without any special fabric includes / excludes. > > ompi_info lists openib as a valid fabric: > $ ompi_info |grep openib > MCA btl: openib (MCA v2.0, API v2.0, Component v1.4.1) > > Thanks for any insight, > > Mike > = > Mike Hanby > mha...@uab.edu > Information Systems Specialist II > IT HPCS / Research Computing > > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] hostfiles
On Feb 4, 2010, at 7:55 PM, Ralph Castain wrote: > Take a look at orte/mca/rmaps/seq - you can select it with -mca rmaps seq > > I believe it is documented ...if it isn't, can it be added to the man page? It might be a common mpirun / hostfile question...? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] DMTCP: Checkpoint-Restart solution for OpenMPI
On Jan 31, 2010, at 10:39 PM, Kapil Arya wrote: > DMTCP also supports a dmtcpaware interface (application-initiated > checkpoints), and numerous other features. At this time, DMTCP > supports only the use of Ethernet (TCP/IP) and shared memory for > transport. We are looking at supporting the Infiniband transport layer > in the future. It sounds like you have taken a virtualized approach to intercepting network calls, etc. Is that right? If so, it would be interesting to see what the performance impact is on some of the OS-bypass / high-performance networks. Previous efforts have taken big performance hits and run into interesting challenges (e.g., can't know the state of the hardware NIC, even if you intercept all the function calls). > Finally, a bit of history. DMTCP began with a goal of checkpointing > distributed desktop applications. We recognize the fine > checkpoint-restart solution that already exists in OpenMPI: > checkpoint-restart service on top of BLCR. We offer DMTCP as an > alternative for some unusual situations, such as when the end user > does not have privilege to add the BLCR kernel module. We are eager > to gain feedback from the OpenMPI community. Have you looked at our plugin capabilities? BLCR is just a plugin to us -- we can support others. Is it worthwhile / possible to hook your technology in via Open MPI plugins? Josh did some great work to make it pretty extensible. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] Anybody built a working 1.4.1 on Solaris 8 (Sparc)?
Terry Dontje wrote > We haven't tried Solaris 8 in quite some time. However, for your first > issue did you include the --enable-heterogeneous option on your > configure command? > > Since you are mix IA-32 and SPARC nodes you'll want to include this so > the endian issue doesn't bite you. Added that on the configure, rebuilt, installed, and now the examples work. Any thoughts on the Forte compiler issue? This is not quite as pressing now that the gcc version works, and most of the computation will be on the remote nodes anyway. Still, the Forte compilers should generate faster code than gcc, and I would prefer to use them if possible. Thanks, David Mathog mat...@caltech.edu Manager, Sequence Analysis Facility, Biology Division, Caltech
Re: [OMPI users] DMTCP: Checkpoint-Restart solution for OpenMPI
On Feb 5, 2010, at 6:40 PM, Gene Cooperman wrote: > You're correct that we take a virtualized approach by intercepting network > calls, etc. However, we purposely never intercept any frequently > called system calls. So, for example, we never intercept a call > to read() or to write() in TCP/IP, as part of our core design principles. > Instead, we use things like the proc filesytem and the use of system calls > to find the offset in an open file descriptor. Coolio. > We would love the opportunity to work with you on a demonstration for > the high-performance networks that you mention. Can you suggest an > MPI code and the appropriate hardware testbed on which we could get an > account and run? Any MPI code should do -- even something as simple as a pass-the-message-around-in-a-ring app. If you can checkpoint and restart it, that's a good start. As for high-speed networks, any iWARP, IB, or Myrinet based network should do. iWARP+IB use the OpenFabrics verbs API; Myrinet networks use the MX API. AFAIK, neither of them export counters through /sys or /proc. > We are aware of your plugin facilities and that in addition to BLCR, > other checkpointers can also integrate with it. And of course, we have the > highest respect for BLCR. We think that at this time, it is best to > continue exploring both approaches. One clarification -- our plugin interfaces were not designed specifically to support BLCR. They were designed to support generic checkpointing facilities. Of course, we only had a few mind when they were designed, so it's possible that they might need to be extended if yours is different than at least the general model that we envisioned. But all things are do-able. I just mention this if you wish to pursue the inside-Open-MPI plugins approach. Of course, staying outside of Open MPI is advantageous from a portability point of view. > Although we haven't looked so closely at the plugin facility, we had > assumed that it always interoperates with the OpenMPI checkpoint-restart > service developed by Joshua Hursey (for which we also have very > high respect). Ya, he's a smart guy. But don't say it too loud or he'll get a big ego! ;-) > Our understanding was that the OpenMPI checkpoint-restart > service arranges to halt all MPI messages, and then it calls BLCR > for checkpointing on the local host. The short answer is "yes". The longer answer is that Josh designed a few different types of plugins -- some of quiescing a job, some for back-end checkpointer support, etc. Hence, one is not directly dependent on the other. I believe that he has written some papers about this... ah, here's one of them (you may have seen this already?): http://www.open-mpi.org/papers/hpdc-2009/ > DMTCP tries to do the job of both Josh's checkpoint-restart service > and also BLCR, and it does it all transparently by working at the TCP/IP > socket > level. So, we simply run: > dmtcp_checkpoint mpirun ./hello_mpi > dmtcp_command --checkpoint > ./dmtcp_restart_script.sh > (The file QUICK-START in DMTCP has a few more details.) > Hence, we don't use the OpenMPI checkpoint-restart service or its plugin > interface, since we're already able to do the distributed checkpointing > directly. If it were important, we could modify DMTCP to be called > by the plugin, and to do checkpointing only on the local host. I guess that's what I was asking about -- if you thought it would be worthwhile to do that: have your checkpoint service be called by Open MPI. In this way, you'd use Open MPI's infrastructure to invoke your single-process-checkpointer underneath. I'm guessing there are advantages and disadvantages to both. > Also, as a side comment, DMTCP was already working with OpenMPI 1.2, but then > later versions of OpenMPI started using more sophisticated system calls. > By then, we were already working through different tasks, and it has > taken us this long to come back to OpenMPI and properly support it again > through our virtualized approach (properly handling the multiple > ptys of OpenMPI, etc.). Gotcha. FWIW, we don't checkpoint the run-time system in Open MPI -- we only checkpoint the MPI processes. Upon restart, we rebuild the run-time system and then launch the "restart" phase in the MPI processes. In this way, we avoided a lot of special case code and took advantage of much of the infrastructure that we already had. This could probably be construed as an advantage to working in the plugin system of Open MPI -- you'd pretty much be isolated from using more complex system calls, etc. Indeed, that was one of Josh's primary design goals: separate the "quiesce" phase from the "checkpoint" phase because they really are two different things, and they don't necessarily have to be related. That being said, a disadvantage of this approach is that that work (i.e., the plugin -- not the actual back-end checkpointer) then becomes specifically tied to Open MPI. Wha
