Re: [OMPI users] IO performance
On 03/02/2012, Tom Rosmond wrote: > Recently the organization I work for bought a modest sized Linux cluster > for running large atmospheric data assimilation systems. In my > experience a glaring problem with systems of this kind is poor IO > performance. Typically they have 2 types of network: 1) A high speed, > low latency, e.g. Infiniband, network dedicated to MPI communications, > and, 2) A lower speed network, e.g 1Gb or 10Gb ethernet, for IO. On > clusters this second network is usually the basis for a global parallel > file system (GPFS), through which nearly all IO traffic must pass. Not necessarily. You can use the Infiniband network for storage very easily, and this is commonly done. Indeed, you can increase your bandwidth for storage by using the two ports of the Infiniband adapters, allocating even and odd compute nodes to different storage subnets. All you need do is have an Infiniband card (or cards) in your storage server, or to put in dedicated storage routers (such as Panasas do). GPFS is not the only kid on the block. There are other filesystems out there - such as Panasas and Lustre. And indeed what is wrong with having a properly balanced RAID array and an NFS server with an Infiniband interface (or interfaces). The 'glaring problem' is not inherent in clusters.
Re: [OMPI users] mpi_abort_delay
On Feb 3, 2012, at 11:45 PM, Aurélien Bouteiller wrote: > The mpi_abort_delay parameters does deadlock the aborting process, as to ease > the manual attachment of gdb, only if the code does indeed call MPI_Abort. > > Is there any similar command line trick to keep alive a process that is > exiting with assert() (or better with segfault etc) ? Not within Open MPI, no. I think if you want to catch those things, you'll need to install your own signal handlers. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] IO performance
On 02/03/2012 01:46 PM, Tom Rosmond wrote: Recently the organization I work for bought a modest sized Linux cluster for running large atmospheric data assimilation systems. In my experience a glaring problem with systems of this kind is poor IO performance. Typically they have 2 types of network: 1) A high speed, (we are biased given that we design, build, and sell these types of high performance storage) Couldn't agree more, high performance in storage systems is often mostly/completely overlooked, and when not, poorly designed/implemented. Its rare we see good implementations. low latency, e.g. Infiniband, network dedicated to MPI communications, and, 2) A lower speed network, e.g 1Gb or 10Gb ethernet, for IO. On clusters this second network is usually the basis for a global parallel file system (GPFS), through which nearly all IO traffic must pass. So the IO performance of applications such as ours is completely dependent on the speed of the GPFS, and therefore on the network hardware it uses. Ouch. 10GbE should be pretty reasonable for high performance traffic, as long as your servers have multiple ports that can be accessed in parallel, and that they are capable of driving data at these rates. When people do focus on the issues mentioned, they usually solve the first of these, and their design to handle the second is broken. We have seen that a cluster with a GPFS based on a 1Gb network is painfully slow for our applications, and of course with a 10Gb network is much better. Therefore we are making the case to the IT staff that all our systems should have GPFS running on 10Gb networks. Some of them have a hard time accepting this, since they don't really understand the requirements of our applications. Well, yes. I don't expect someone used to dealing with desktops, mail servers, etc. to really grasp why you need a very high bandwidth very low latency network for messages or storage. If you see their stacked switch architectures, you might shudder to think that these are replicated in cluster environments (they are when IT has a hand in designing them). With all of this, here is my MPI related question. I recently added an option to use MPI-IO to do the heavy IO lifting in our applications. I would like to know what the relative importance of the dedicated MPI network vis-a-vis the GPFS network for typical MPIIO collective reads and writes. I assume there must be some hand-off of data between the networks during the process, but how is it done, and are there any rules to help understand it. Any insights would be welcome. We'd recommend all storage be used over the fast networks (10GbE and Infiniband). Fairly painless to configure. This said, I guess one of my concerns would be the design of the storage servers ... I am guessing they may not be able to even come close to saturating a 10GbE connection, never mind an Infiniband connection. All this said, you might find an audience that can help with the operational aspects of this better on the beowulf list than here on the OpenMPI list. Certainly OpenMPI list is a great place to discuss the MPI-IO side of things, but the performance side of the system design (outside of MPI) might be better located on a different list. Just a suggestion ... T. Rosmond P.S. I am running with Open-mpi 1.4.2. -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics Inc. email: land...@scalableinformatics.com web : http://scalableinformatics.com http://scalableinformatics.com/sicluster phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615
Re: [OMPI users] O-MPI Support for heterogeneous (Windows / Linux) clusters
On Feb 3, 2012, at 8:54 AM, James Torossian wrote: > Hi all, > > I noticed about 18 months ago in the user archives that the question of > heterogeneous support for clusters of Windows / Linux machines was raised, > and at that time this functionality was not supported. > > Is this still the case? Are there any near term plans in this direction? I don't think anyone plans to make that work, especially since there is a very simple workaround available. Just install one of the free ssh server packages for Windows and you can use the rsh/ssh launcher. Follow the instructions on the OMPI FAQ for hetero operations and all should work fine. > > Also … still related …. > > Has it been possible to build O-MPI for Windows (with perhaps reduced > capabilities) to use ssh under Cygwin rather than WMI? We are only after a > small subset of MPI functionality. Afraid I don't work that area…perhaps Shiqing could answer it? > > Thanks, > Jim > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] How to join intercommunicators?
Look at MPI_Comm_join On Feb 3, 2012, at 2:12 PM, Mateus Augusto wrote: > Hello, > > I have a process that creates other processes at different times (using > MPI_Comm_spawn). After, the initial process (creator of other processes) must > receive mesagens of created processes. However, there are one > intercommunicator for each created process. > The question is: how to unite all the intercommunicator in a single > communicator and receive the mesagens from this single comunicator? > > Thanks > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
[OMPI users] Open MPI-1.5.4 --with-mpi-f90-size=large Compilation Error
Dear Open MPI users, I want to compile Open MPI-1.5.4 beta on my macbook pro (with GCC-4.6.2/Gfortran-4.6.2 installed) with the option; " configure --with-mpi-f90-size=large ". The configuration script runs ok but during compilation I get the following warnings which after a limit of 25 turn into an error. FC mpi_scatterv_f90.lo mpi_scatterv_f90.f90:17.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:55.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) FC mpi_sendrecv_f90.lo mpi_scatterv_f90.f90:93.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:131.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:169.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:207.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:245.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:283.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:321.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:359.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:397.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:435.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:473.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:511.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:549.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:587.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:625.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:663.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:701.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:739.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:777.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:815.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:853.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:891.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) mpi_scatterv_f90.f90:929.12: print *, "Open MPI WARNING: You are calling MPI_SCATTERV with incorrect sendc 1 Error: Unterminated character constant beginning at (1) Fatal Error: Error count reached limit of 25. make[4]: *** [mpi_scatterv_f90.lo] Error 1 make[4]: *** Waiting for unfinished jobs.
