Ralph, For the following openMPI job submission:
qsub -q short.q -V -pe make 84 -b y mpirun -np 84 --prefix /hpc/apps/mpi/openmpi/1.10.1/ --hetero-nodes --mca btl ^sm --mca plm_base_verbose 5 /hpc/home/lanew/mpi/openmpi/a_1_10_1.out I have some more information on this issue. All the server daemons are started without error and before I ever see the [csclprd3-5-2:10512] [[42154,0],0] plm:base:receive got update_proc_state for job [42154,1] [csclprd3-6-12:30667] *** Process received signal *** [csclprd3-6-12:30667] Signal: Bus error (7) qrsh throws the following error for various nodes taking part in the openmpi compute ring: unable to write to file /tmp/285507.1.short.q/qrsh_error: No space left on device[csclprd3-4-3:08052] [[24964,0],17] plm:rsh: using "/opt/sge/bin/lx-amd64/qrsh -inherit -nostdin -V -verbose" for launching Does each and every node taking part in the openMPI compute ring need to write to a temporary directory? Also, the submit node for this cluster has a soft open files limit of 1024 and a hard open files limit of 4096. All compute nodes have a hard/soft open files limit of 4096. -Bill L. ________________________________ From: users [users-boun...@open-mpi.org] on behalf of Ralph Castain [r...@open-mpi.org] Sent: Thursday, March 17, 2016 6:02 PM To: Open MPI Users Subject: Re: [OMPI users] OpenMPI 1.10.1 *ix hard/soft open files limits >= 4096 still required? Yeah, it looks like something is wrong with the mmap backend for some reason. It gets used by both vader and sm, so no help there. I’m afraid I’ll have to defer to Nathan from here as he is more familiar with it than I. On Mar 17, 2016, at 4:55 PM, Lane, William <william.l...@cshs.org<mailto:william.l...@cshs.org>> wrote: I ran OpenMPI using the "-mca btl ^vader" switch Ralph recommended and I'm still getting the same errors qsub -q short.q -V -pe make 206 -b y mpirun -np 206 --prefix /hpc/apps/mpi/openmpi/1.10.1/ --hetero-nodes --mca btl ^vader --mca plm_base_verbose 5 /hpc/home/lanew/mpi/openmpi/a_1_10_1.out [csclprd3-5-2:10512] [[42154,0],0] plm:base:receive got update_proc_state for job [42154,1] [csclprd3-6-12:30667] *** Process received signal *** [csclprd3-6-12:30667] Signal: Bus error (7) [csclprd3-6-12:30667] Signal code: Non-existant physical address (2) [csclprd3-6-12:30667] Failing at address: 0x2b1b18a2d000 [csclprd3-6-12:30667] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1b0e06c500] [csclprd3-6-12:30667] [ 1] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_shmem_mmap.so(+0x1524)[0x2b1b0f5fd524] [csclprd3-6-12:30667] [ 2] /hpc/apps/mpi/openmpi/1.10.1/lib/libmca_common_sm.so.4(mca_common_sm_module_create_and_attach+0x56)[0x2b1b124daab6] [csclprd3-6-12:30667] [ 3] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_btl_sm.so(+0x39cb)[0x2b1b12d749cb] [csclprd3-6-12:30667] [ 4] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_btl_sm.so(+0x3f2a)[0x2b1b12d74f2a] [csclprd3-6-12:30667] [ 5] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_btl_base_select+0x117)[0x2b1b0ddfdb07] [csclprd3-6-12:30667] [ 6] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x2b1b126de7b2] [csclprd3-6-12:30667] [ 7] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_bml_base_init+0x99)[0x2b1b0ddfd309] [csclprd3-6-12:30667] [ 8] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_pml_ob1.so(+0x538c)[0x2b1b133a138c] [csclprd3-6-12:30667] [ 9] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_pml_base_select+0x1e0)[0x2b1b0de0e780] [csclprd3-6-12:30667] [10] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(ompi_mpi_init+0x51d)[0x2b1b0ddc017d] [csclprd3-6-12:30667] [11] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(MPI_Init+0x170)[0x2b1b0dddf820] [csclprd3-6-12:30667] [12] /hpc/home/lanew/mpi/openmpi/a_1_10_1.out[0x400ad0] [csclprd3-6-12:30667] [13] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1b0e298cdd] [csclprd3-6-12:30667] [14] /hpc/home/lanew/mpi/openmpi/a_1_10_1.out[0x400999] [csclprd3-6-12:30667] *** End of error message *** -Bill L. ________________________________ From: users [users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] on behalf of Lane, William [william.l...@cshs.org<mailto:william.l...@cshs.org>] Sent: Thursday, March 17, 2016 4:49 PM To: Open MPI Users Subject: Re: [OMPI users] OpenMPI 1.10.1 *ix hard/soft open files limits >= 4096 still required? I apologize Ralph, I forgot to include my command line for invoking OpenMPI on SoGE: qsub -q short.q -V -pe make 87 -b y mpirun -np 87 --prefix /hpc/apps/mpi/openmpi/1.10.1/ --hetero-nodes --mca btl ^sm --mca plm_base_verbose 5 /hpc/home/lanew/mpi/openmpi/a_1_10_1.out a_1_10_1.out is my OpenMPI test code binary compiled under OpenMPI 1.10.1. Thanks for the quick response! -Bill L. ________________________________ From: users [users-boun...@open-mpi.org<mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain [r...@open-mpi.org<mailto:r...@open-mpi.org>] Sent: Thursday, March 17, 2016 4:44 PM To: Open MPI Users Subject: Re: [OMPI users] OpenMPI 1.10.1 *ix hard/soft open files limits >= 4096 still required? No, that shouldn’t be the issue any more - and that isn’t what the backtrace indicates. It looks instead like there was a problem with the shared memory backing file on a remote node, and that caused the vader shared memory BTL to segfault. Try turning vader off and see if that helps - I’m not sure what you are using, but maybe “-mca btl ^vader” will suffice Nathan - any other suggestions? On Mar 17, 2016, at 4:40 PM, Lane, William <william.l...@cshs.org<mailto:william.l...@cshs.org>> wrote: I remember years ago, OpenMPI (version 1.3.3) required the hard/soft open files limits be >= 4096 in order to function when large numbers of slots were requested (with 1.3.3 this was at roughly 85 slots). Is this requirement still present for OpenMPI versions 1.10.1 and greater? I'm having some issues now with OpenMPI version 1.10.1 that remind me of the issues I had w/1.3.3 where OpenMPI worked fine as long as I don't request too many slots. When I look at the ulimits -a (soft limit) I see: open files (-n) 1024 Ulimits -Ha (hard limit) gives: open files (-n) 4096 I'm getting errors of the form: [csclprd3-5-5:15248] [[40732,0],0] plm:base:receive got update_proc_state for job [40732,1] [csclprd3-6-12:30567] *** Process received signal *** [csclprd3-6-12:30567] Signal: Bus error (7) [csclprd3-6-12:30567] Signal code: Non-existant physical address (2) [csclprd3-6-12:30567] Failing at address: 0x2b3d19f72000 [csclprd3-6-12:30568] *** Process received signal *** [csclprd3-6-12:30567] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b3d0f71f500] [csclprd3-6-12:30567] [ 1] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_shmem_mmap.so(+0x1524)[0x2b3d10cb0524] [csclprd3-6-12:30567] [ 2] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_btl_vader.so(+0x3674)[0x2b3d18494674] [csclprd3-6-12:30567] [ 3] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_btl_base_select+0x117)[0x2b3d0f4b0b07] [csclprd3-6-12:30567] [ 4] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x2b3d13d917b2] [csclprd3-6-12:30567] [ 5] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_bml_base_init+0x99)[0x2b3d0f4b0309] [csclprd3-6-12:30567] [ 6] /hpc/apps/mpi/openmpi/1.10.1/lib/openmpi/mca_pml_ob1.so(+0x538c)[0x2b3d18ac238c] [csclprd3-6-12:30567] [ 7] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(mca_pml_base_select+0x1e0)[0x2b3d0f4c1780] [csclprd3-6-12:30567] [ 8] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(ompi_mpi_init+0x51d)[0x2b3d0f47317d] [csclprd3-6-12:30567] [ 9] /hpc/apps/mpi/openmpi/1.10.1/lib/libmpi.so.12(MPI_Init+0x170)[0x2b3d0f492820] [csclprd3-6-12:30567] [10] /hpc/home/lanew/mpi/openmpi/a_1_10_1.out[0x400ad0] [csclprd3-6-12:30567] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b3d0f94bcdd] [csclprd3-6-12:30567] [12] /hpc/home/lanew/mpi/openmpi/a_1_10_1.out[0x400999] [csclprd3-6-12:30567] *** End of error message *** Ugh. Bill L. IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation. _______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/03/28746.php IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation. IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation._______________________________________________ users mailing list us...@open-mpi.org<mailto:us...@open-mpi.org> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users Link to this post: http://www.open-mpi.org/community/lists/users/2016/03/28749.php IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation.
