The line
Signal code: Address not mapped (1)
indicates that there is probably a mismatch between the runtime
library and the linked version. Make sure that you link the program
and run it using the same installation base. Are the libraries in /
usr/mpi/fsl_openmpi_gcc_1.2.6 the same you used at link time?
On Sep 19, 2008, at 2:42 PM, Daniel Hansen wrote:
I work for a supercomputing organization and we just installed the
latest version of rocks/centos on our cluster. We compiled openmpi
from source to customize it for our purposes. Since switching we
have receive messages from users about errors, segfaults, etc. that
we didn't see before. Here is one such segfault message that I
don't have enough knowledge to figure out or even have a clue about
what is going on. Here it is:
[m4b-1-8:11830] *** Process received signal ***
[m4b-1-8:11830] Signal: Segmentation fault (11)
[m4b-1-8:11830] Signal code: Address not mapped (1)
[m4b-1-8:11830] Failing at address: 0x2abcdff475b0
[m4b-1-8:11830] [ 0] /lib64/libpthread.so.0 [0x33e8c0de70]
[m4b-1-8:11830] [ 1] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_btl_sm_send+0xf1) [0x2aaaaab541d1]
[m4b-1-8:11830] [ 2] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_send_request_start_copy+0x15e) [0x2aaaaaba0e2e]
[m4b-1-8:11830] [ 3] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_isend+0x217) [0x2aaaaab9be37]
[m4b-1-8:11830] [ 4] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_sendrecv_actual+0xda) [0x2aaaaab5acaa]
[m4b-1-8:11830] [ 5] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_barrier_intra_bruck+0x9f) [0x2aaaaab5f81f]
[m4b-1-8:11830] [ 6] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(PMPI_Barrier+0x6f) [0x2aaaaab1eadf]
[m4b-1-8:11830] [ 7] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug(main+0x5d9) [0x413179]
[m4b-1-8:11830] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4)
[0x33e841d8a4]
[m4b-1-8:11830] [ 9] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug [0x404109]
[m4b-1-8:11830] *** End of error message ***
[m4b-1-9:11772] *** Process received signal ***
[m4b-1-9:11772] Signal: Segmentation fault (11)
[m4b-1-9:11772] Signal code: Address not mapped (1)
[m4b-1-9:11772] Failing at address: 0x2abcdff475b0
[m4b-1-9:11772] [ 0] /lib64/libpthread.so.0 [0x302380de70]
[m4b-1-9:11772] [ 1] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_btl_sm_send+0xf1) [0x2aaaaab541d1]
[m4b-1-9:11772] [ 2] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_send_request_start_copy+0x15e) [0x2aaaaaba0e2e]
[m4b-1-9:11772] [ 3] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_isend+0x217) [0x2aaaaab9be37]
[m4b-1-9:11772] [ 4] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_sendrecv_actual+0xda) [0x2aaaaab5acaa]
[m4b-1-9:11772] [ 5] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_barrier_intra_bruck+0x9f) [0x2aaaaab5f81f]
[m4b-1-9:11772] [ 6] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(PMPI_Barrier+0x6f) [0x2aaaaab1eadf]
[m4b-1-9:11772] [ 7] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug(main+0x5d9) [0x413179]
[m4b-1-9:11772] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4)
[0x302301d8a4]
[m4b-1-9:11772] [ 9] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug [0x404109]
[m4b-1-9:11772] *** End of error message ***
[m4b-1-7i][0,1,7][btl_tcp_endpoint.c:
572:mca_btl_tcp_endpoint_complete_connect] connect() failed with
errno=111
[m4b-1-7i][0,1,8][btl_tcp_endpoint.c:
572:mca_btl_tcp_endpoint_complete_connect] connect() failed with
errno=111
[m4b-1-7i][0,1,9][btl_tcp_frag.c:202:mca_btl_tcp_frag_recv]
mca_btl_tcp_frag_recv: readv failed with errno=104
[m4b-1-7i][0,1,9][btl_tcp_endpoint.c:
572:mca_btl_tcp_endpoint_complete_connect] connect() failed with
errno=111
[m4b-1-9:11773] *** Process received signal ***
[m4b-1-9:11773] Signal: Segmentation fault (11)
[m4b-1-9:11773] Signal code: Address not mapped (1)
[m4b-1-9:11773] Failing at address: 0x2abcdff475b0
[m4b-1-9:11773] [ 0] /lib64/libpthread.so.0 [0x302380de70]
[m4b-1-9:11773] [ 1] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_btl_sm_send+0xf1) [0x2aaaaab541d1]
[m4b-1-9:11773] [ 2] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_send_request_start_copy+0x15e) [0x2aaaaaba0e2e]
[m4b-1-9:11773] [ 3] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(mca_pml_ob1_isend+0x217) [0x2aaaaab9be37]
[m4b-1-9:11773] [ 4] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_sendrecv_actual+0xda) [0x2aaaaab5acaa]
[m4b-1-9:11773] [ 5] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(ompi_coll_tuned_barrier_intra_bruck+0x9f) [0x2aaaaab5f81f]
[m4b-1-9:11773] [ 6] /usr/mpi/fsl_openmpi_gcc_1.2.6/lib/libmpi.so.0
(PMPI_Barrier+0x6f) [0x2aaaaab1eadf]
[m4b-1-9:11773] [ 7] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug(main+0x5d9) [0x413179]
[m4b-1-9:11773] [ 8] /lib64/libc.so.6(__libc_start_main+0xf4)
[0x302301d8a4]
[m4b-1-9:11773] [ 9] /fslhome/wshuai/compute/for_Shuai2/
mpi_md_bgo_twham_12sept08_debug [0x404109]
[m4b-1-9:11773] *** End of error message ***
orterun noticed that job rank 0 with PID 12338 on node m4b-1-10i
exited on signal 15 (Terminated).
Can someone give me some clues as to what is going wrong here or
possibly point me in the right direction? Is there something I or
the user can do to get more informative error messages? The user
mentioned that this particular program ran fine before we upgraded
to the current openmpi version, and that he can't find any bugs in
his code.
Thanks for your help,
Daniel Hansen
Systems Administrator
BYU Fulton Supercomputing Lab
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
