Re: [OMPI users] OpenMPI and Port Range

[Simon Hammond]

On 31/08/2007, Lev Givon  wrote:
> Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > I have a patch for this, but I never felt a real need for it, so I
> > never push it in the trunk. I'm not completely convinced that we need
> > it, except in some really strange situations (read grid). Why do you
> > need a port range ? For avoiding firewalls ?

We are planning on using OpenMPI as the basis for running MPI jobs
across a series of workstations overnight. The workstations are locked
down so that only a small number of ports are available for use. If we
try to use anything else its disaster.

Unfortunately this is really an organizational policy above anything
else and its very difficult to get it to change.


Si Hammond
University of Warwick

> >
> >Thanks,
> >  george.
>
> I imagine that allowing for more security-conscious firewall
> configurations would be the main motivation (although I suspect that
> there are more folks who run MPI on tightly coupled clusters linked by
> a secure/private network than on grids of machines spread across an
> insecure network).
>
> L.G.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OpenMPI and Port Range

[Gleb Natapov]

On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> On 31/08/2007, Lev Givon  wrote:
> > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > I have a patch for this, but I never felt a real need for it, so I
> > > never push it in the trunk. I'm not completely convinced that we need
> > > it, except in some really strange situations (read grid). Why do you
> > > need a port range ? For avoiding firewalls ?
> 
> We are planning on using OpenMPI as the basis for running MPI jobs
> across a series of workstations overnight. The workstations are locked
> down so that only a small number of ports are available for use. If we
> try to use anything else its disaster.
> 
> Unfortunately this is really an organizational policy above anything
> else and its very difficult to get it to change.
> 
> 
As workaround you can write application that will bind to all ports that
are not allowed to be used by MPI before running MPI job.

--
Gleb.

Re: [OMPI users] OpenMPI and Port Range

[Simon Hammond]

On 31/08/2007, Gleb Natapov  wrote:
> On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> > On 31/08/2007, Lev Givon  wrote:
> > > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > > I have a patch for this, but I never felt a real need for it, so I
> > > > never push it in the trunk. I'm not completely convinced that we need
> > > > it, except in some really strange situations (read grid). Why do you
> > > > need a port range ? For avoiding firewalls ?
> >
> > We are planning on using OpenMPI as the basis for running MPI jobs
> > across a series of workstations overnight. The workstations are locked
> > down so that only a small number of ports are available for use. If we
> > try to use anything else its disaster.
> >
> > Unfortunately this is really an organizational policy above anything
> > else and its very difficult to get it to change.
> >
> >
> As workaround you can write application that will bind to all ports that
> are not allowed to be used by MPI before running MPI job.

Sounds very drastic, thanks for the advice. I'll give it a go. Do you
think it might be easy to add this to the source code at sometime
though?

Thanks for your advice guys,

Si Hammond
University of Warwick

>
> --
> Gleb.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OpenMPI and Port Range

[Gleb Natapov]

On Fri, Aug 31, 2007 at 08:17:36AM +0100, Simon Hammond wrote:
> On 31/08/2007, Gleb Natapov  wrote:
> > On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> > > On 31/08/2007, Lev Givon  wrote:
> > > > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > > > I have a patch for this, but I never felt a real need for it, so I
> > > > > never push it in the trunk. I'm not completely convinced that we need
> > > > > it, except in some really strange situations (read grid). Why do you
> > > > > need a port range ? For avoiding firewalls ?
> > >
> > > We are planning on using OpenMPI as the basis for running MPI jobs
> > > across a series of workstations overnight. The workstations are locked
> > > down so that only a small number of ports are available for use. If we
> > > try to use anything else its disaster.
> > >
> > > Unfortunately this is really an organizational policy above anything
> > > else and its very difficult to get it to change.
> > >
> > >
> > As workaround you can write application that will bind to all ports that
> > are not allowed to be used by MPI before running MPI job.
> 
> Sounds very drastic, thanks for the advice. I'll give it a go. Do you
> think it might be easy to add this to the source code at sometime
> though?
>
It just workaround. Proper solution would be of cause adding an option for this.

--
Gleb.

Re: [OMPI users] OpenMPI and Port Range

[Sven Stork]

On Friday 31 August 2007 09:07, Gleb Natapov wrote:
> On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> > On 31/08/2007, Lev Givon  wrote:
> > > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > > I have a patch for this, but I never felt a real need for it, so I
> > > > never push it in the trunk. I'm not completely convinced that we need
> > > > it, except in some really strange situations (read grid). Why do you
> > > > need a port range ? For avoiding firewalls ?
> > 
> > We are planning on using OpenMPI as the basis for running MPI jobs
> > across a series of workstations overnight. The workstations are locked
> > down so that only a small number of ports are available for use. If we
> > try to use anything else its disaster.
> > 
> > Unfortunately this is really an organizational policy above anything
> > else and its very difficult to get it to change.
> > 
> > 
> As workaround you can write application that will bind to all ports that
> are not allowed to be used by MPI before running MPI job.

Another option could be (if that match your policy) to limit the dynamic port 
range that is used by your OS. By this all application (unless they ask for 
an specific port) will get ports in this limited port range. If so the 
following link might be interesting for you:

http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

-- Sven 

> --
>   Gleb.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 

Re: [OMPI users] OpenMPI and Port Range

[Gleb Natapov]

On Fri, Aug 31, 2007 at 10:49:10AM +0200, Sven Stork wrote:
> On Friday 31 August 2007 09:07, Gleb Natapov wrote:
> > On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> > > On 31/08/2007, Lev Givon  wrote:
> > > > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > > > I have a patch for this, but I never felt a real need for it, so I
> > > > > never push it in the trunk. I'm not completely convinced that we need
> > > > > it, except in some really strange situations (read grid). Why do you
> > > > > need a port range ? For avoiding firewalls ?
> > > 
> > > We are planning on using OpenMPI as the basis for running MPI jobs
> > > across a series of workstations overnight. The workstations are locked
> > > down so that only a small number of ports are available for use. If we
> > > try to use anything else its disaster.
> > > 
> > > Unfortunately this is really an organizational policy above anything
> > > else and its very difficult to get it to change.
> > > 
> > > 
> > As workaround you can write application that will bind to all ports that
> > are not allowed to be used by MPI before running MPI job.
> 
> Another option could be (if that match your policy) to limit the dynamic port 
> range that is used by your OS. By this all application (unless they ask for 
> an specific port) will get ports in this limited port range. If so the 
> following link might be interesting for you:
> 
> http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
> 
I was sure it is possible to set a port range on Linux, but didn't know how.
This is much better workaround.

--
Gleb.

Re: [OMPI users] OpenMPI and Port Range

[Simon Hammond]

On 31/08/2007, Gleb Natapov  wrote:
> On Fri, Aug 31, 2007 at 10:49:10AM +0200, Sven Stork wrote:
> > On Friday 31 August 2007 09:07, Gleb Natapov wrote:
> > > On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:
> > > > On 31/08/2007, Lev Givon  wrote:
> > > > > Received from George Bosilca on Thu, Aug 30, 2007 at 07:42:52PM EDT:
> > > > > > I have a patch for this, but I never felt a real need for it, so I
> > > > > > never push it in the trunk. I'm not completely convinced that we 
> > > > > > need
> > > > > > it, except in some really strange situations (read grid). Why do you
> > > > > > need a port range ? For avoiding firewalls ?
> > > >
> > > > We are planning on using OpenMPI as the basis for running MPI jobs
> > > > across a series of workstations overnight. The workstations are locked
> > > > down so that only a small number of ports are available for use. If we
> > > > try to use anything else its disaster.
> > > >
> > > > Unfortunately this is really an organizational policy above anything
> > > > else and its very difficult to get it to change.
> > > >
> > > >
> > > As workaround you can write application that will bind to all ports that
> > > are not allowed to be used by MPI before running MPI job.
> >
> > Another option could be (if that match your policy) to limit the dynamic 
> > port
> > range that is used by your OS. By this all application (unless they ask for
> > an specific port) will get ports in this limited port range. If so the
> > following link might be interesting for you:
> >
> > http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
> >
> I was sure it is possible to set a port range on Linux, but didn't know how.
> This is much better workaround.

Thanks guys, I'll give this a try.


Si Hammond

>
> --
> Gleb.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OpenMPI and Port Range

[Jeff Squyres]

George --

What do you say about committing your patch?


On Aug 31, 2007, at 7:06 AM, Simon Hammond wrote:


On 31/08/2007, Gleb Natapov  wrote:

On Fri, Aug 31, 2007 at 10:49:10AM +0200, Sven Stork wrote:

On Friday 31 August 2007 09:07, Gleb Natapov wrote:

On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:

On 31/08/2007, Lev Givon  wrote:
Received from George Bosilca on Thu, Aug 30, 2007 at  
07:42:52PM EDT:
I have a patch for this, but I never felt a real need for it,  
so I
never push it in the trunk. I'm not completely convinced that  
we need
it, except in some really strange situations (read grid). Why  
do you

need a port range ? For avoiding firewalls ?


We are planning on using OpenMPI as the basis for running MPI jobs
across a series of workstations overnight. The workstations are  
locked
down so that only a small number of ports are available for  
use. If we

try to use anything else its disaster.

Unfortunately this is really an organizational policy above  
anything

else and its very difficult to get it to change.


As workaround you can write application that will bind to all  
ports that

are not allowed to be used by MPI before running MPI job.


Another option could be (if that match your policy) to limit the  
dynamic port
range that is used by your OS. By this all application (unless  
they ask for
an specific port) will get ports in this limited port range. If  
so the

following link might be interesting for you:

http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html

I was sure it is possible to set a port range on Linux, but didn't  
know how.

This is much better workaround.


Thanks guys, I'll give this a try.


Si Hammond



--
Gleb.
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users



--
Jeff Squyres
Cisco Systems

Re: [OMPI users] buildsystem / adio-lustre-mpich2-v02.patch

[Jeff Squyres]

On Aug 30, 2007, at 4:24 PM, Robert Latham wrote:

I'm presently trying to add lustre support to open-mpi's romio  
using this
patch http://ft.ornl.gov/projects/io/src/adio-lustre-mpich2- 
v02.patch.


It basically applies, only a few C files have been renamed in open- 
mpi, but

the autotools build system gives me headaches.


Boy tell me about it (says the guy who's job it is to work on it).


In hindsight, renaming the files may not have been a bit over the  
top.  We were trying to have strict adherence to our file/symbol  
prefix internal naming rules to guarantee collision-free code  
(especially since we're based on plugins).


But it was very necessary to integrate ROMIO into our autotools setup  
for reasons too boring to go into here.


I know that Brian (who has been working with Rob) has mentioned that  
he has some ideas about simplifying our ROMIO integration, but I  
don't think he's had any cycles to work on them (which is also why I  
suspect he has not answered this post, too).


Fine, adding a similar entry for lustre is easy, but now we need  
to define

BUILD_XFS. So where does this come from?
There is romio/romio/Makefile.in


I don't know where you got BUILD_XFS.  In MPICH2, we don't use that
symbol anywhere.


BUILD_UFS_FALSE = @BUILD_UFS_FALSE@
BUILD_UFS_TRUE = @BUILD_UFS_TRUE@
BUILD_XFS_FALSE = @BUILD_XFS_FALSE@
BUILD_XFS_TRUE = @BUILD_XFS_TRUE@
CC = @CC@


These look like OpenMPI additions.  Perhaps they can chime in.


They probably are.


Now lets assume I eventually find the proper autotools files to
patch lustre support in, how can I distribute that patch?


Distribute changes to the primary sources.  i.e. fix configure.in or
Makefile.am and let autotools regenerate the necessary files.
ROMIO's got an AC_PREREQ in there, so things can't go horribly bad


FWIW, OMPI has strict requirements on the versions of autotools that  
it uses:


http://www.open-mpi.org/svn/building.php


In principle I would have to distribute a patch that also patches
the auto-generated configure, automake.in, etc files.


As you understand already, that's untennable.


Any plans for a sane build system?


I'm not proud of how ROMIO's configure.in has evolved over the years.
not proud at all.  But I can gaurantee you that the alternatives are
worse.  Don't speak to me of scons and cmake!

I hope the MPICH2 perspective on ROMIO's build system helps a bit, but
I think the integration with OpenMPI may have changed things somewhat.


Continual re-integration of ROMIO is definitely a logistics problem  
that we have not solved.  And it's becoming a bigger problem.  :-(


Normally, we're quite open to accepting patches to Open MPI to put  
them into the main distribution to ease the whole "millions of MPI  
distros" issue, but with ROMIO it becomes quite difficult because we  
have to source from Argonne's copy.  Trying to manage what patches  
need to go in is already quite difficult because:


- ROMIO is not on our release schedule
- OMPI adds its own integration patches to ROMIO
- All the OMPI developers have other work to do ;-)

Adding 3rd party patches in there for something that we already know  
is complex and understaffed has unfortunately been a low priority.  :-(


One thing that may make things a little better is that Brian recently  
integrated some work onto the OMPI trunk that allows ROMIO to be  
built outside of OMPI.  Hence, if you have a standalone ROMIO, OMPI  
can use it.  I don't know the details (i.e., if you can still use  
mpi.h / MPI_Request / MPI_Test / MPI_Wait like you can with the  
default OMPI ROMIO integration) -- Brian will have to chime in here...


So I don't know what the real solution is here -- I'm just trying to  
give some of the OMPI perspective.  Suggestions are welcome.   
Probably the best solution would be someone to volunteer to actually  
spend the cycles to maintain ROMIO in Open MPI (I am pretty sure that  
Brian simply does not have them)...


--
Jeff Squyres
Cisco Systems

Re: [OMPI users] OpenMPI and Port Range

[James Conway]

The patch for limiting the port range used by OpenMPI sounds useful.  
This *is* an issue with firewalls. For a dedicated cluster behind a  
firewall that's ok, but for harnessing available hardware (eg, Mac  
OSX systems) in the wee hours its not. Gets my vote!


James Conway

On Aug 31, 2007, at 8:26 AM, Jeff Squyres wrote:


George --

What do you say about committing your patch?


On Aug 31, 2007, at 7:06 AM, Simon Hammond wrote:


On 31/08/2007, Gleb Natapov  wrote:

On Fri, Aug 31, 2007 at 10:49:10AM +0200, Sven Stork wrote:

On Friday 31 August 2007 09:07, Gleb Natapov wrote:

On Fri, Aug 31, 2007 at 08:04:00AM +0100, Simon Hammond wrote:

On 31/08/2007, Lev Givon  wrote:

Received from George Bosilca on Thu, Aug 30, 2007 at
07:42:52PM EDT:

I have a patch for this, but I never felt a real need for it,
so I never push it in the trunk. I'm not completely  
convinced that
we need it, except in some really strange situations (read  
grid). Why

do you need a port range ? For avoiding firewalls ?

...
--
James Conway, PhD.,
Department of Structural Biology
University of Pittsburgh School of Medicine
Biomedical Science Tower 3, Room 2047
3501 5th Ave
Pittsburgh, PA 15260
U.S.A.
Phone: +1-412-383-9847
Fax:   +1-412-648-8998
Email: jxc...@pitt.edu
Web:    (under construction)
--

Re: [OMPI users] OpenMPI and Port Range

[Jeff Squyres]

On Aug 31, 2007, at 9:07 AM, James Conway wrote:


The patch for limiting the port range used by OpenMPI sounds useful.
This *is* an issue with firewalls. For a dedicated cluster behind a
firewall that's ok, but for harnessing available hardware (eg, Mac
OSX systems) in the wee hours its not. Gets my vote!


Note that restricting TCP ports will not allow OMPI to be used  
between clusters that have a firewall between them.  There are other  
issues involved, such as public IP address mirrors, etc.


George's patch *may* allow OMPI to be used when a per-host firewall  
restricts which ports may be used, as long as all hosts are allowed  
unfettered access to the incoming port range.


--
Jeff Squyres
Cisco Systems

[OMPI users] Bug in MPI_Reduce/MPI_Comm_split?

[Marco Sbrighi]

Dear Open MPI developers,

I'm using Open MPI 1.2.2 over OFED 1.1 on an 680 nodes dual Opteron dual
core Linux cluster. Of course, with Infiniband interconnect. 
During the execution of big jobs (greater than 128 processes) I've
experimented slow down in performances and deadlock in collective MPI
operations. The job processes terminates often issuing "RETRY EXCEEDED
ERROR", of course if the  btl_openib_ib_timeout is properly set.  
Yes, this kind of error seems to be related to the fabric, but more or
less half of the MPI processes are incurring in timeout. 
In order to do a better investigation on that behaviour, I've tried to
do some "constrained" tests using SKaMPI, but it is quite difficult to
insulate a single collective operation using SKaMPI. In fact despite the
SKaMPI script can contain only a request for (say) a Reduce, with many
communicator sizes, the SKaMPI code will make also a lot of bcast,
alltoall etc. by itself.
So I've tried to use an hand made piece of code, in order to do "only" a
repeated collective operation at a time.
The code is attached to this message, the file is named
collect_noparms.c.
What is happened when I've tried to run this code is reported here:

..

011 - 011 - 039 NOOT START
000 - 000 of 38 - 655360  0.00
[node1049:11804] *** Process received signal ***
[node1049:11804] Signal: Segmentation fault (11)
[node1049:11804] Signal code: Address not mapped (1)
[node1049:11804] Failing at address: 0x18
035 - 035 - 039 NOOT START
000 - 000 of 38 - 786432  0.00
[node1049:11804] [ 0] /lib64/tls/libpthread.so.0 [0x2a964db420]
000 - 000 of 38 - 917504  0.00
[node1049:11804] [ 1] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0 
[0x2a9573fa18]
[node1049:11804] [ 2] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0 
[0x2a9573f639]
[node1049:11804] [ 3] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(mca_btl_sm_send+0x122)
 [0x2a9573f5e1]
[node1049:11804] [ 4] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0 
[0x2a957acac6]
[node1049:11804] [ 5] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(mca_pml_ob1_send_request_start_copy+0x303)
 [0x2a957ace52]
[node1049:11804] [ 6] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0 
[0x2a957a2788]
[node1049:11804] [ 7] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0 
[0x2a957a251c]
[node1049:11804] [ 8] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(mca_pml_ob1_send+0x2e2)
 [0x2a957a2d9e]
[node1049:11804] [ 9] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(ompi_coll_tuned_reduce_generic+0x651)
 [0x2a95751621]
[node1049:11804] [10] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(ompi_coll_tuned_reduce_intra_pipeline+0x176)
 [0x2a95751bff]
[node1049:11804] [11] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(ompi_coll_tuned_reduce_intra_dec_fixed+0x3f4)
 [0x2a957475f6]
[node1049:11804] [12] 
/cineca/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/libmpi.so.0(PMPI_Reduce+0x3a6)
 [0x2a9570a076]
[node1049:11804] [13] 
/bcx/usercin/asm0/mpptools/mpitools/debug/src/collect_noparms_bc.x(reduce+0x3e) 
[0x404e64]
[node1049:11804] [14] 
/bcx/usercin/asm0/mpptools/mpitools/debug/src/collect_noparms_bc.x(main+0x620) 
[0x404c8e]
[node1049:11804] [15] /lib64/tls/libc.so.6(__libc_start_main+0xdb) 
[0x2a966004bb]
[node1049:11804] [16] 
/bcx/usercin/asm0/mpptools/mpitools/debug/src/collect_noparms_bc.x [0x40448a]
[node1049:11804] *** End of error message ***

...

the behaviour is the same, more or less identical, using either
Infiniband or Gigabit interconnect. If I use another MPI implementation
(say MVAPICH), all goes right.
Then I've compiled both my code and Open MPI using gcc 3.4.4 with
bounds-checking, compiler debugging flags, without OMPI memory
manager ... the behaviour is identical but now I've the line were the
SIGSEGV is trapped:



gdb collect_noparms_bc.x core.11580
GNU gdb Red Hat Linux (6.3.0.0-1.96rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db 
library "/lib64/tls/libthread_db.so.1".


warning: core file may not match specified executable file.
Core was generated by 
`/bcx/usercin/asm0/mpptools/mpitools/debug/src/collect_noparms_bc.x'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from 
/prod/openmpi/1.2.2/mr/gnu3.4-bc_no_memory_mgr_dbg/lib/l

[OMPI users] MPI Migration

[Mauricio Vacas]

Hello everyone,

I am doing research on the different migration mechanisms available for
MPI processes. I had a couple of questions I was hoping you guys you
clarify for me concerning this:

1. Are there other approaches to MPI migration that do not involve
checkpoint/restart mechanisms (e.g. LAM/MPI with BLCR, MPICH-V with BLCR
and Condor)?
2. MPI-2 allows for dynamic process spawning, is it possible to spawn an
MPI process on a different node and transfer the process state to the
newly created process as long as you keep account of the new location
for the other MPI processes?
3. MOSIX and Charm++ perform process migration with MPI processes, are
there any problems associated with using these systems for MPI process
migration?

Thanks!
Mauricio

Re: [OMPI users] buildsystem / adio-lustre-mpich2-v02.patch

[Bernd Schubert]

Robert Latham wrote:

> On Sun, Aug 26, 2007 at 06:44:18PM +0200, Bernd Schubert wrote:
>> I'm presently trying to add lustre support to open-mpi's romio using this
>> patch http://ft.ornl.gov/projects/io/src/adio-lustre-mpich2-v02.patch.
>> 
>> It basically applies, only a few C files have been renamed in open-mpi,
>> but the autotools build system gives me headaches.
> 
> Boy tell me about it (says the guy who's job it is to work on it).

I don't know much about it myself, I only read a discussion about it on the
lustre mailing list.

> 
>> Fine, adding a similar entry for lustre is easy, but now we need to
>> define BUILD_XFS. So where does this come from?
>> There is romio/romio/Makefile.in
> 
> I don't know where you got BUILD_XFS.  In MPICH2, we don't use that
> symbol anywhere.

This was supposed to be BUILD_LUSTRE, taken as example from BUILD_XFS. I was
a bit overworked that weekend.

>  
>> BUILD_UFS_FALSE = @BUILD_UFS_FALSE@
>> BUILD_UFS_TRUE = @BUILD_UFS_TRUE@
>> BUILD_XFS_FALSE = @BUILD_XFS_FALSE@
>> BUILD_XFS_TRUE = @BUILD_XFS_TRUE@
>> CC = @CC@
> 
> These look like OpenMPI additions.  Perhaps they can chime in.
> 
> In MPICH2-land we build up FILE_SYS_DIRS in configure.in, and then do
> a "for dir in $FILE_SYS_DIRS", with each directory contributing some
> .o files to the io library.  Hey, it works.

Yes, the patch from ft.ornl.gov also does it that way, well its for mpich2.

> 
>> Not a single line about filesystems, *grumble*.
> 
> Again speaking about MPICH2 land, all fileystem logic (such as it is)
> lives in configure.in.  it's regrettably scattered amongst some
> architecture-specific sections.

To make it short, I could get everything to compile and to work (though more
or less a brute force way), but I'm still not pleased with the performance.
So far I didn't have the time yet to investigate the reason why the
performance is still low, though, I suspect openmpi/ofed problems, because
as long as I run the computations on a single system without
networking/infiniband the i/o performance is rather good.
I first need to figure out why writing to nfs-exported lustre is slow and
then I will care about MPI. Well, I already know the reason for the nfs
problem, now I need to write a proper patch. I guess I will start to work
on the MPI problem on Wednesday.

> 
>> Now lets assume I eventually find the proper autotools files to
>> patch lustre support in, how can I distribute that patch?
> 
> Distribute changes to the primary sources.  i.e. fix configure.in or
> Makefile.am and let autotools regenerate the necessary files.
> ROMIO's got an AC_PREREQ in there, so things can't go horribly bad

So I need to write an explanation what to do after applying the patch? Who
actually reads documentation ;) ?

> 
> (like in the old days when people would try to autoconf ROMIO's
> ancient autoconf-1.7 era configure with autoconf-2.13 -- you heard me.
> You think it's bad now?  Why, back in '00 you were lucky if you ... ok
> you get the idea. i'll quit with the old man bit.)
> 
>> In principle I would have to distribute a patch that also patches
>> the auto-generated configure, automake.in, etc files.
> 
> As you understand already, that's untennable.

Well, the original patch does it that way.

> 
>> Any plans for a sane build system?
> 
> I'm not proud of how ROMIO's configure.in has evolved over the years.
> not proud at all.  But I can gaurantee you that the alternatives are
> worse.  Don't speak to me of scons and cmake!

I would have suggested cmake. Whats so bad about it?

> 
> I hope the MPICH2 perspective on ROMIO's build system helps a bit, but
> I think the integration with OpenMPI may have changed things somewhat.
> 
> Good luck

Thanks!

Cheers,
Bernd


PS: The patch is here:
http://www.pci.uni-heidelberg.de/tc/usr/bernd/downloads/lustre/openmpi/

Apply it and then run in ompi/mca/io/romio/romio/

aclocal
autoheader
autoconf
libtoolize --force
automake -a -c