Hi,
Here is the output of a core file generated during a segmentation fault
observed during a collective call (using openib):
#0 0x0000000000000000 in ?? ()
(gdb) where
#0 0x0000000000000000 in ?? ()
#1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
(openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700, byte_len=18) at
btl_openib_component.c:2881 #2 0x00002aedbc4e25e2 in handle_wc
(device=0x19024ac0, cq=0, wc=0x7ffff279ce90) at
btl_openib_component.c:3178 #3 0x00002aedbc4e2e9d in poll_device
(device=0x19024ac0, count=2) at btl_openib_component.c:3318 #4
0x00002aedbc4e34b8 in progress_one_device (device=0x19024ac0) at
btl_openib_component.c:3426 #5 0x00002aedbc4e3561 in
btl_openib_component_progress () at btl_openib_component.c:3451 #6
0x00002aedb8b22ab8 in opal_progress () at runtime/opal_progress.c:207 #7
0x00002aedb859f497 in opal_condition_wait (c=0x2aedb888ccc0,
m=0x2aedb888cd20) at ../opal/threads/condition.h:99 #8 0x00002aedb859fa31
in ompi_request_default_wait_all (count=2, requests=0x7ffff279d0e0,
statuses=0x0) at request/req_wait.c:262 #9 0x00002aedbd7559ad in
ompi_coll_tuned_allreduce_intra_recursivedoubling (sbuf=0x7ffff279d444,
rbuf=0x7ffff279d440, count=1, dtype=0x6788220, op=0x6787a20,
comm=0x19d81ff0, module=0x19d82b20) at coll_tuned_allreduce.c:223
#10 0x00002aedbd7514f7 in ompi_coll_tuned_allreduce_intra_dec_fixed
(sbuf=0x7ffff279d444, rbuf=0x7ffff279d440, count=1, dtype=0x6788220,
op=0x6787a20, comm=0x19d81ff0, module=0x19d82b20) at
coll_tuned_decision_fixed.c:63
#11 0x00002aedb85c7792 in PMPI_Allreduce (sendbuf=0x7ffff279d444,
recvbuf=0x7ffff279d440, count=1, datatype=0x6788220, op=0x6787a20,
comm=0x19d81ff0) at pallreduce.c:102 #12 0x0000000004387dbf in
FEMTown::MPI::Allreduce (sendbuf=0x7ffff279d444, recvbuf=0x7ffff279d440,
count=1, datatype=0x6788220, op=0x6787a20, comm=0x19d81ff0) at
stubs.cpp:626 #13 0x0000000004058be8 in FEMTown::Domain::align (itf=
{<FEMTown::Boost::shared_base_ptr<FEMTown::Domain::Interface>>
= {_vptr.shared_base_ptr = 0x7ffff279d620, ptr_ = {px = 0x199942a4, pn =
{pi_ = 0x6}}},<No data fields>}) at interface.cpp:371
#14 0x00000000040cb858 in FEMTown::Field::detail::align_itfs_and_neighbhors
(dim=2, set={px = 0x7ffff279d780, pn = {pi_ = 0x2f279d640}},
check_info=@0x7ffff279d7f0) at check.cpp:63 #15 0x00000000040cbfa8 in
FEMTown::Field::align_elements (set={px = 0x7ffff279d950, pn = {pi_ =
0x66e08d0}}, check_info=@0x7ffff279d7f0) at check.cpp:159 #16
0x00000000039acdd4 in PyField_align_elements (self=0x0,
args=0x2aaab0765050, kwds=0x19d2e950) at check.cpp:31 #17
0x0000000001fbf76d in FEMTown::Main::ExErrCatch<_object* (*)(_object*,
_object*, _object*)>::exec<_object> (this=0x7ffff279dc20, s=0x0,
po1=0x2aaab0765050, po2=0x19d2e950) at
/home/qa/svntop/femtown/modules/main/py/exception.hpp:463
#18 0x00000000039acc82 in PyField_align_elements_ewrap (self=0x0,
args=0x2aaab0765050, kwds=0x19d2e950) at check.cpp:39 #19
0x00000000044093a0 in PyEval_EvalFrameEx (f=0x19b52e90, throwflag=<value
optimized out>) at Python/ceval.c:3921 #20 0x000000000440aae9 in
PyEval_EvalCodeEx (co=0x2aaab754ad50, globals=<value optimized out>,
locals=<value optimized out>, args=0x3, argcount=1, kws=0x19ace4a0,
kwcount=2, defs=0x2aaab75e4800, defcount=2, closure=0x0) at
Python/ceval.c:2968
#21 0x0000000004408f58 in PyEval_EvalFrameEx (f=0x19ace2d0,
throwflag=<value optimized out>) at Python/ceval.c:3802 #22
0x000000000440aae9 in PyEval_EvalCodeEx (co=0x2aaab7550120, globals=<value
optimized out>, locals=<value optimized out>, args=0x7, argcount=1,
kws=0x19acc418, kwcount=3, defs=0x2aaab759e958, defcount=6, closure=0x0)
at Python/ceval.c:2968
#23 0x0000000004408f58 in PyEval_EvalFrameEx (f=0x19acc1c0,
throwflag=<value optimized out>) at Python/ceval.c:3802 #24
0x000000000440aae9 in PyEval_EvalCodeEx (co=0x2aaab8b5e738, globals=<value
optimized out>, locals=<value optimized out>, args=0x6, argcount=1,
kws=0x19abd328, kwcount=5, defs=0x2aaab891b7e8, defcount=3, closure=0x0)
at Python/ceval.c:2968
#25 0x0000000004408f58 in PyEval_EvalFrameEx (f=0x19abcea0,
throwflag=<value optimized out>) at Python/ceval.c:3802 #26
0x000000000440aae9 in PyEval_EvalCodeEx (co=0x2aaab3eb4198, globals=<value
optimized out>, locals=<value optimized out>, args=0xb, argcount=1,
kws=0x19a89df0, kwcount=10, defs=0x0, defcount=0, closure=0x0) at
Python/ceval.c:2968
#27 0x0000000004408f58 in PyEval_EvalFrameEx (f=0x19a89c40,
throwflag=<value optimized out>) at Python/ceval.c:3802 #28
0x000000000440aae9 in PyEval_EvalCodeEx (co=0x2aaab3eb4288, globals=<value
optimized out>, locals=<value optimized out>, args=0x1, argcount=0,
kws=0x19a89330, kwcount=0, defs=0x2aaab8b66668, defcount=1, closure=0x0)
at Python/ceval.c:2968
#29 0x0000000004408f58 in PyEval_EvalFrameEx (f=0x19a891b0,
throwflag=<value optimized out>) at Python/ceval.c:3802 #30
0x000000000440aae9 in PyEval_EvalCodeEx (co=0x2aaab8b6a738, globals=<value
optimized out>, locals=<value optimized out>, args=0x0, argcount=0,
kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at
Python/ceval.c:2968
#31 0x000000000440ac02 in PyEval_EvalCode (co=0x1902f9b0, globals=0x0,
locals=0x190d9700) at Python/ceval.c:522 #32 0x000000000442853c in
PyRun_StringFlags (str=0x192fd3d8 "DIRECT.Actran.main()", start=<value
optimized out>, globals=0x192213d0, locals=0x192213d0, flags=0x0) at
Python/pythonrun.c:1335 #33 0x0000000004429690 in PyRun_SimpleStringFlags
(command=0x192fd3d8 "DIRECT.Actran.main()", flags=0x0) at
Python/pythonrun.c:957 #34 0x0000000001fa1cf9 in
FEMTown::Python::FEMPy::run_application (this=0x7ffff279f650) at
fempy.cpp:873 #35 0x000000000434ce99 in FEMTown::Main::Batch::run
(this=0x7ffff279f650) at batch.cpp:374 #36 0x0000000001f9aa25 in main
(argc=8, argv=0x7ffff279fa48) at main.cpp:10 (gdb) f 1
#1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
(openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700, byte_len=18) at
btl_openib_component.c:2881 2881 reg->cbfunc(
&openib_btl->super, hdr->tag, des, reg->cbdata ); Current language: auto;
currently c
(gdb)
#1 0x00002aedbc4e05f4 in btl_openib_handle_incoming
(openib_btl=0x1902f9b0, ep=0x1908a1c0, frag=0x190d9700, byte_len=18) at
btl_openib_component.c:2881 2881 reg->cbfunc(
&openib_btl->super, hdr->tag, des, reg->cbdata ); (gdb) l
2876
2877 if(OPAL_LIKELY(!(is_credit_msg = is_credit_message(frag)))) {
2878 /* call registered callback */
2879 mca_btl_active_message_callback_t* reg;
2880 reg = mca_btl_base_active_message_trigger + hdr->tag;
2881 reg->cbfunc(&openib_btl->super, hdr->tag, des, reg->cbdata
); 2882 if(MCA_BTL_OPENIB_RDMA_FRAG(frag)) {
2883 cqp = (hdr->credits>> 11)& 0x0f;
2884 hdr->credits&= 0x87ff;
2885 } else {
Regards,
Eloi
On Friday 16 July 2010 16:01:02 Eloi Gaudry wrote:
Hi Edgar,
The only difference I could observed was that the segmentation fault
appeared sometimes later during the parallel computation.
I'm running out of idea here. I wish I could use the "--mca coll tuned"
with "--mca self,sm,tcp" so that I could check that the issue is not
somehow limited to the tuned collective routines.
Thanks,
Eloi
On Thursday 15 July 2010 17:24:24 Edgar Gabriel wrote:
On 7/15/2010 10:18 AM, Eloi Gaudry wrote:
hi edgar,
thanks for the tips, I'm gonna try this option as well. the
segmentation fault i'm observing always happened during a collective
communication indeed... does it basically switch all collective
communication to basic mode, right ?
sorry for my ignorance, but what's a NCA ?
sorry, I meant to type HCA (InifinBand networking card)
Thanks
Edgar
thanks,
éloi
On Thursday 15 July 2010 16:20:54 Edgar Gabriel wrote:
you could try first to use the algorithms in the basic module, e.g.
mpirun -np x --mca coll basic ./mytest
and see whether this makes a difference. I used to observe sometimes
a (similar ?) problem in the openib btl triggered from the tuned
collective component, in cases where the ofed libraries were
installed but no NCA was found on a node. It used to work however
with the basic component.
Thanks
Edgar
On 7/15/2010 3:08 AM, Eloi Gaudry wrote:
hi Rolf,
unfortunately, i couldn't get rid of that annoying segmentation
fault when selecting another bcast algorithm. i'm now going to
replace MPI_Bcast with a naive implementation (using MPI_Send and
MPI_Recv) and see if that helps.
regards,
éloi
On Wednesday 14 July 2010 10:59:53 Eloi Gaudry wrote:
Hi Rolf,
thanks for your input. You're right, I miss the
coll_tuned_use_dynamic_rules option.
I'll check if I the segmentation fault disappears when using the
basic bcast linear algorithm using the proper command line you
provided.
Regards,
Eloi
On Tuesday 13 July 2010 20:39:59 Rolf vandeVaart wrote:
Hi Eloi:
To select the different bcast algorithms, you need to add an
extra mca parameter that tells the library to use dynamic
selection. --mca coll_tuned_use_dynamic_rules 1
One way to make sure you are typing this in correctly is to use
it with ompi_info. Do the following:
ompi_info -mca coll_tuned_use_dynamic_rules 1 --param coll
You should see lots of output with all the different algorithms
that can be selected for the various collectives.
Therefore, you need this:
--mca coll_tuned_use_dynamic_rules 1 --mca
coll_tuned_bcast_algorithm 1
Rolf
On 07/13/10 11:28, Eloi Gaudry wrote:
Hi,
I've found that "--mca coll_tuned_bcast_algorithm 1" allowed to
switch to the basic linear algorithm. Anyway whatever the
algorithm used, the segmentation fault remains.
Does anyone could give some advice on ways to diagnose the issue
I'm facing ?
Regards,
Eloi
On Monday 12 July 2010 10:53:58 Eloi Gaudry wrote:
Hi,
I'm focusing on the MPI_Bcast routine that seems to randomly
segfault when using the openib btl. I'd like to know if there
is any way to make OpenMPI switch to a different algorithm
than the default one being selected for MPI_Bcast.
Thanks for your help,
Eloi
On Friday 02 July 2010 11:06:52 Eloi Gaudry wrote:
Hi,
I'm observing a random segmentation fault during an internode
parallel computation involving the openib btl and
OpenMPI-1.4.2 (the same issue can be observed with
OpenMPI-1.3.3).
mpirun (Open MPI) 1.4.2
Report bugs to http://www.open-mpi.org/community/help/
[pbn08:02624] *** Process received signal ***
[pbn08:02624] Signal: Segmentation fault (11)
[pbn08:02624] Signal code: Address not mapped (1)
[pbn08:02624] Failing at address: (nil)
[pbn08:02624] [ 0] /lib64/libpthread.so.0 [0x349540e4c0]
[pbn08:02624] *** End of error message ***
sh: line 1: 2624 Segmentation fault
\/share\/hpc3\/actran_suite\/Actran_11\.0\.rc2\.41872\/RedHatE
L\ -5 \/ x 86 _6 4\ /bin\/actranpy_mp
'--apl=/share/hpc3/actran_suite/Actran_11.0.rc2.41872/RedHatEL
-5 /x 86 _ 64 /A c tran_11.0.rc2.41872'
'--inputfile=/work/st25652/LSF_130073_0_47696_0/Case1_3Dreal_m
4_ n2 .d a t'
'--scratch=/scratch/st25652/LSF_130073_0_47696_0/scratch'
'--mem=3200' '--threads=1' '--errorlevel=FATAL' '--t_max=0.1'
'--parallel=domain'
If I choose not to use the openib btl (by using --mca btl
self,sm,tcp on the command line, for instance), I don't
encounter any problem and the parallel computation runs
flawlessly.
I would like to get some help to be able:
- to diagnose the issue I'm facing with the openib btl
- understand why this issue is observed only when using the
openib btl and not when using self,sm,tcp
Any help would be very much appreciated.
The outputs of ompi_info and the configure scripts of OpenMPI
are enclosed to this email, and some information on the
infiniband drivers as well.
Here is the command line used when launching a parallel
computation
using infiniband:
path_to_openmpi/bin/mpirun -np $NPROCESS --hostfile
host.list --mca
btl openib,sm,self,tcp --display-map --verbose --version
--mca mpi_warn_on_fork 0 --mca btl_openib_want_fork_support 0
[...]
and the command line used if not using infiniband:
path_to_openmpi/bin/mpirun -np $NPROCESS --hostfile
host.list --mca
btl self,sm,tcp --display-map --verbose --version --mca
mpi_warn_on_fork 0 --mca btl_openib_want_fork_support 0 [...]
Thanks,
Eloi
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
