Hi Josh,
Thanks for your reply. Actually the reason for hang was missing
blcr library in LD_LIBRARY_PATH.
After setting it right, checkpoint was working but as you
mentioned before, datatype error is coming along with, and hence restart
is not working.
a) The errors coming is
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[n0:12674] *** An error occurred in MPI_Barrier
[n0:12674] *** on communicator MPI_COMM_WORLD
[n0:12674] *** MPI_ERR_BUFFER: invalid buffer pointer
[n0:12674] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
[n0:12674] crcp:bkmrk: drain_message_copy_remove(): Datatype copy failed
(1)
[n0:12674] crcp:bkmrk: drain_check_recv(): Datatype copy failed (1)
[n0:12674] crcp:bkmrk: pml_recv(): Failed trying to find a drained
message. ---------- This should never happen ----------
(crcp_bkmrk_pml.c:2411)
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
b) The other error i am getting while checkpointing is segmentation
fault. This is independent of the previous error and the scenario is bit
different.
I have two nodes, one with infiniband and other with TCP. I am
running simple mpirun with no option of selecting/deselecting btl's
leaving openmpi to decide at runtime.
The error i am getting is as follows:
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
[n5:29211] *** Process received signal ***
[n5:29211] Signal: Segmentation fault (11)
[n5:29211] Signal code: Address not mapped (1)
[n5:29211] Failing at address: (nil)
[n5:29211] [ 0] /lib64/tls/libpthread.so.0 [0x399e80c4f0]
[n5:29211] [ 1] /usr/lib64/libibverbs.so.1(ibv_destroy_srq+0)
[0x3496108fb0]
[n5:29211] [ 2]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_btl_openib.so
[0x2a979369ed]
[n5:29211] [ 3]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_btl_openib.so
[0x2a979376b5]
[n5:29211] [ 4]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_bml_r2.so
[0x2a97186a33]
[n5:29211] [ 5]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_pml_ob1.so
[0x2a96f68e5d]
[n5:29211] [ 6]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_pml_crcpw.so
[0x2a96e638d5]
[n5:29211] [ 7]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/libmpi.so.0(ompi_cr_coord+0x127)
[0x2a95591127]
[n5:29211] [ 8]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/libopen-pal.so.0(opal_cr_inc_core+0x33)
[0x2a95858403]
[n5:29211] [ 9]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_snapc_full.so
[0x2a965432b1]
[n5:29211] [10]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/libopen-pal.so.0(opal_cr_test_if_checkpoint_ready+0x52)
[0x2a95857662]
[n5:29211] [11]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/libmpi.so.0
[0x2a9558e13b]
[n5:29211] [12]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_coll_tuned.so
[0x2a98421c12]
[n5:29211] [13]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/openmpi/mca_coll_tuned.so
[0x2a9842a00e]
[n5:29211] [14]
/home/syssoft/alap/fts/openmpi-1.3.1/ft_thread_install/lib/libmpi.so.0(PMPI_Barrier+0x140)
[0x2a955a4af0]
[n5:29211] [15] ./a.out(main+0x64) [0x4009bc]
[n5:29211] [16] /lib64/tls/libc.so.6(__libc_start_main+0xdb)
[0x399e11c3fb]
[n5:29211] [17] ./a.out [0x4008ca]
[n5:29211] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 1 with PID 29211 on node n5 exited on
signal 11 (Segmentation fault).
--------------------------------------------------------------------------
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Soon, this error goes away, if i force mpirun to use tcp for
communication using mca parameters and then error a) starts coming which
is related with some datatype handling during checkpoint.
Regards
Neeraj Chourasia
Member of Technical Staff
Computational Research Laboratories Limited
(A wholly Owned Subsidiary of TATA SONS Ltd)
P: +91.9225520634
Josh Hursey <jjhur...@open-mpi.org>
Sent by: users-boun...@open-mpi.org
04/28/2009 12:34 AM
Please respond to
Open MPI Users <us...@open-mpi.org>
To
Open MPI Users <us...@open-mpi.org>
cc
Subject
Re: [OMPI users] Checkpointing hangs with OpenMPI-1.3.1
I still have not been able to reproduce the hang, but I'm still
looking into it.
I did commit a fix for the datatype copy error that I mentioned
(r21080 in the Open MPI trunk, and it is in the pipeline for v1.3).
Can you put in a print statement before MPI_Finalize, then try the
program again? I am wondering if the problem is not with the
MPI_Barrier, but MPI_Finalize. I wonder if one (or both) of the
processes enter MPI_Finalize while a checkpoint is occurring.
Unfortunately, I have not tested the MPI_Finalize scenario in a long
time, but will put that on my todo list.
Cheers,
Josh
On Apr 27, 2009, at 9:48 AM, Josh Hursey wrote:
> Sorry for the long delay to respond.
>
> It is a bit odd that the hang does not occur when running on only
> one host. I suspect that is more due to timing than anything else.
>
> I am not able to reproduce the hang at the moment, but I do get an
> occasional datatype copy error which could be symptomatic of a
> related problem. I'll dig into this a bit more this week and let you
> know when I have a fix and if I can reproduce the hang.
>
> Thanks for the bug report.
>
> Cheers,
> Josh
>
> On Apr 10, 2009, at 4:56 AM, nee...@crlindia.com wrote:
>
>>
>> Dear All,
>>
>> I am trying to checkpoint a test application using openmpi-1.3.1,
>> but fails to do so, when run multiple process on different nodes.
>>
>> Checkpointing runs fine, if process is running on the same node
>> along with mpirun process. But the moment i launch MPI process from
>> different node, it hangs.
>>
>> ex.
>> mpirun -np 2 ./test (will checkpoint fine using ompi-checkpoint -
>> v <mpirun_pid> )
>> but
>> mpirun -np 2 -H host1 ./test (Checkpointing will hang)
>>
>> Similarly
>> mpirun -np 2 -H localhost,host1 ./test would still hangs while
>> checkpointing.
>>
>> Please find the output which comes while checkpointing
>>
>> --------------
>> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx----------------------------
>> [n0:01596] orte_checkpoint: Checkpointing...
>> [n0:01596] PID 1514
>> [n0:01596] Connected to Mpirun [[11946,0],0]
>> [n0:01596] orte_checkpoint: notify_hnp: Contact Head Node Process
>> PID 1514
>> [n0:01596] orte_checkpoint: notify_hnp: Requested a checkpoint of
>> jobid [INVALID]
>> [n0:01596] orte_checkpoint: hnp_receiver: Receive a command message.
>> [n0:01596] orte_checkpoint: hnp_receiver: Status Update.
>> [n0:01596] Requested - Global Snapshot Reference:
>> (null)
>> [n0:01596] orte_checkpoint: hnp_receiver: Receive a command message.
>> [n0:01596] orte_checkpoint: hnp_receiver: Status Update.
>> [n0:01596] Pending - Global Snapshot Reference:
>> (null)
>> [n0:01596] orte_checkpoint: hnp_receiver: Receive a command message.
>> [n0:01596] orte_checkpoint: hnp_receiver: Status Update.
>> [n0:01596] Running - Global Snapshot Reference:
>> (null)
>>
>> Note: It hangs here
>>
>> ------------------------------
>> *******************************---------------------
>>
>> Command used to launch program is
>>
>> /usr/local/openmpi-1.3.1/install/bin/mpirun -np 2 -H n5 -am ft-
>> enable-cr --mca btl tcp,self a.out
>>
>> And the dummy program is pretty simple as follows
>>
>> #include<time.h>
>> #include<stdio.h>
>> #include<mpi.h>
>>
>>
>> #define LIMIT 10000000
>>
>> main(int argc,char ** argv)
>> {
>> int i;
>>
>> int my_rank; /* Rank of process */
>> int np; /* Number of process */
>>
>>
>> MPI_Init(&argc,&argv);
>> MPI_Comm_rank(MPI_COMM_WORLD,&my_rank);
>> MPI_Comm_size(MPI_COMM_WORLD, &np);
>>
>>
>> for(i=0; i<=LIMIT; i++)
>> {
>> printf("n HELLO %d",i);
>> //sleep(10);
>> MPI_Barrier(MPI_COMM_WORLD);
>> }
>> MPI_Finalize();
>> }
>>
>>
>>
>> Let me know, what could be the error. I feel there is the error in
>> MPI process coordination.
>>
>> Regards
>>
>>
>> Neeraj Chourasia
>> Member of Technical Staff
>> Computational Research Laboratories Limited
>> (A wholly Owned Subsidiary of TATA SONS Ltd)
>> P: +91.9890003757
>>
>> =====-----=====-----===== Notice: The information contained in this
>> e-mail message and/or attachments to it may contain confidential or
>> privileged information. If you are not the intended recipient, any
>> dissemination, use, review, distribution, printing or copying of
>> the information contained in this e-mail message and/or attachments
>> to it are strictly prohibited. If you have received this
>> communication in error, please notify us by reply e-mail or
>> telephone and immediately and permanently delete the message and
>> any attachments. Internet communications cannot be guaranteed to be
>> timely, secure, error or virus-free. The sender does not accept
>> liability for any errors or omissions.Thank you =====-----=====-----
>> =====
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments.
Internet communications cannot be guaranteed to be timely,
secure, error or virus-free. The sender does not accept liability
for any errors or omissions.Thank you
=====-----=====-----=====
