Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-02-05 Thread Josh Hursey
This is a bit late in the thread, but I wanted to add one more note. The functionality that made it to v1.6 is fairly basic in terms of C/R support in Open MPI. It supported a global checkpoint write, and (for a time) a simple staged option (I think that is now broken). In the trunk (about 3 year

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-30 Thread Maxime Boissonneault
Le 2013-01-29 21:02, Ralph Castain a écrit : On Jan 28, 2013, at 10:53 AM, Maxime Boissonneault > wrote: While our filesystem and management nodes are on UPS, our compute nodes are not. With one average generic (power/cooling mostly) failure ever

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-30 Thread Constantinos Makassikis
On Wed, Jan 30, 2013 at 3:02 AM, Ralph Castain wrote: > > If your node hardware is the problem, or you decide you do want/need to > pursue an FT solution, then you might look at the OMPI-based solutions from > parties such as http://fault-tolerance.org or the MPICH2 folks. > Just as Ralph said,

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-29 Thread Ralph Castain
On Jan 28, 2013, at 10:53 AM, Maxime Boissonneault wrote: > While our filesystem and management nodes are on UPS, our compute nodes are > not. With one average generic (power/cooling mostly) failure every one or two > months, running for weeks is just asking for trouble. If you add to that >

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread George Bosilca
Based on the paper you linked the answer is quite obvious. The proposed CRFS mechanism supports all of the checkpoint-enabled MPI implementation, thus you just have to go with the one providing and caring about the services you need. George. On Mon, Jan 28, 2013 at 3:46 PM, Maxime Boissonneault

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Maxime Boissonneault
Hi George, The problem here is not the bandwidth, but the number of IOPs. I wrote to the BLCR list, and they confirmed that : "While ideally the checkpoint would be written in sizable chunks, the current code in BLCR will issue a single write operation for each contiguous range of user memory,

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread George Bosilca
At the scale you address you should have no trouble with the C/R if the file system is correctly configured. We get more bandwidth per node out of an NFS over 1Gb/s at 32 nodes. Have you run some parallel benchmarks on your cluster ? George. PS: You can some MPI I/O benchmarks at http://www.mcs.

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Ralph Castain
On Jan 28, 2013, at 10:53 AM, Maxime Boissonneault wrote: > Le 2013-01-28 13:15, Ralph Castain a écrit : >> On Jan 28, 2013, at 9:52 AM, Maxime Boissonneault >> wrote: >> >>> Le 2013-01-28 12:46, Ralph Castain a écrit : On Jan 28, 2013, at 8:25 AM, Maxime Boissonneault wrote: >>>

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Maxime Boissonneault
Le 2013-01-28 13:15, Ralph Castain a écrit : On Jan 28, 2013, at 9:52 AM, Maxime Boissonneault wrote: Le 2013-01-28 12:46, Ralph Castain a écrit : On Jan 28, 2013, at 8:25 AM, Maxime Boissonneault wrote: Hello Ralph, I agree that ideally, someone would implement checkpointing in the appl

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Ralph Castain
On Jan 28, 2013, at 9:52 AM, Maxime Boissonneault wrote: > Le 2013-01-28 12:46, Ralph Castain a écrit : >> On Jan 28, 2013, at 8:25 AM, Maxime Boissonneault >> wrote: >> >>> Hello Ralph, >>> I agree that ideally, someone would implement checkpointing in the >>> application itself, but that

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Maxime Boissonneault
Le 2013-01-28 12:46, Ralph Castain a écrit : On Jan 28, 2013, at 8:25 AM, Maxime Boissonneault wrote: Hello Ralph, I agree that ideally, someone would implement checkpointing in the application itself, but that is not always possible (commercial applications, use of complicated libraries, a

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Ralph Castain
On Jan 28, 2013, at 8:25 AM, Maxime Boissonneault wrote: > Hello Ralph, > I agree that ideally, someone would implement checkpointing in the > application itself, but that is not always possible (commercial applications, > use of complicated libraries, algorithms with no clear progression poi

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Maxime Boissonneault
Hello Ralph, I agree that ideally, someone would implement checkpointing in the application itself, but that is not always possible (commercial applications, use of complicated libraries, algorithms with no clear progression points at which you can interrupt the algorithm and start it back fro

Re: [OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Ralph Castain
Our c/r person has moved on to a different career path, so we may not have anyone who can answer this question. What we can say is that checkpointing at any significant scale will always be a losing proposition. It just takes too long and hammers the file system. People have been working on ext

[OMPI users] Checkpointing an MPI application with OMPI

2013-01-28 Thread Maxime Boissonneault
Hello, I am doing checkpointing tests (with BLCR) with an MPI application compiled with OpenMPI 1.6.3, and I am seeing behaviors that are quite strange. First, some details about the tests : - The only filesystem available on the nodes are 1) one tmpfs, 2) one lustre shared filesystem (tested