I'm running a mixed cluster of Blades (HS21 and HS22 chassis), x3550-M3 and X3550-M4 systems, some of which support hyperthreading, while others don't (specifically the HS21 blades) all on CentOS 6.3 w/SGE.
I have no problems running my simple OpenMPI 1.8.7 test code outside of SGE (with or without the --bind-to core switch, but can only run the jobs within SGE via qrsh on a limited number of slots (4 at most) successfully. The errors are very similar to the ones I was getting running OpenMPI 1.8.5 - 1.8.6 outside of SGE on this same cluster. Strangely, when running the test code outside of SGE w/the --bind-to core switch, mpirun still binds processes to hyperthreading cores. Additionally, the --bind-to core switch prevents the OpenMPI 1.8.7 test code from running at all within SGE (it outputs warnings about missing NUMA libraries reducing performance then exits). We would rather run out OpenMPI jobs from within SGE so that we can get accounting data on OpenMPI jobs for administrative purposes. The orte PE I'm been using seems to meet all the requirements for previous versions of OpenMPI: the allocation rule is fill-up, rather than round-robin (I'm not sure if this makes a difference or not) The value NONE in user_lists and xuser_lists mean enable everybody and exclude nobody. The value of control_slaves must be TRUE; otherwise, qrsh exits with an error message. The value of job_is_first_task must be FALSE or the job launcher consumes a slot. In other words, mpirun itself will count as one of the slots and the job will fail, because only n-1 processes will start. And be sure the queue will make use of the PE that you specified Below is the command line I've been using to generate the errors found in the attached file out.txt: qrsh -V -now yes -pe orte 186 mpirun -np 186 --prefix /hpc/apps/mpi/openmpi/1.8.7/ --mca btl_tcp_if_include eth0 --mca pls_gridengine_verbose 1 /hpc/home/lanew/mpi/openmpi/ProcessColors3 >> out.txt 2>&1 Sorry for the length. Thanks in advance for any help in resolving this nagging issue (I wish we had a homogenous cluster now). -Bill L. IMPORTANT WARNING: This message is intended for the use of the person or entity to which it is addressed and may contain information that is privileged and confidential, the disclosure of which is governed by applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this information is strictly prohibited. Thank you for your cooperation.
--enable-debug switch wasn't recognized outside of SGE executable will run at up to 189 nodes (many more are actually available), the --bind-to core directive doesn't seem to stop mpirun from binding processes to hyperthreads rather than cores also getting core dumps from aborted mpiruns via qrsh and qsub with the --bind-to core switch job doesn't even run, there is no output qrsh -V -now yes -pe orte 186 mpirun -np 186 --prefix /hpc/apps/mpi/openmpi/1.8.7/ --mca btl_tcp_if_include eth0 --mca pls_gridengine_verbose 1 /hpc/home/lanew/mpi/openmpi/ProcessColors3 >> out.txt 2>&1 [csclprd3-4-3:13849] *** Process received signal *** [csclprd3-4-3:13851] *** Process received signal *** [csclprd3-4-3:13851] Signal: Bus error (7) [csclprd3-4-3:13851] Signal code: Non-existant physical address (2) [csclprd3-4-3:13851] Failing at address: 0x2b06d6f56980 [csclprd3-4-3:13853] *** Process received signal *** [csclprd3-4-3:13853] Signal: Bus error (7) [csclprd3-4-3:13853] Signal code: Non-existant physical address (2) [csclprd3-4-3:13853] Failing at address: 0x2b715ac0cf00 [csclprd3-4-3:13857] *** Process received signal *** [csclprd3-4-3:13859] *** Process received signal *** [csclprd3-4-3:13859] Signal: Bus error (7) [csclprd3-4-3:13859] Signal code: Non-existant physical address (2) [csclprd3-4-3:13859] Failing at address: 0x2b9a315cac00 [csclprd3-4-3:13861] *** Process received signal *** [csclprd3-4-3:13863] *** Process received signal *** [csclprd3-4-3:13865] *** Process received signal *** [csclprd3-4-3:13867] *** Process received signal *** [csclprd3-4-3:13867] Signal: Bus error (7) [csclprd3-4-3:13867] Signal code: Non-existant physical address (2) [csclprd3-4-3:13867] Failing at address: 0x2b7f1cab1980 [csclprd3-4-3:13873] *** Process received signal *** [csclprd3-4-3:13875] *** Process received signal *** [csclprd3-4-3:13875] Signal: Bus error (7) [csclprd3-4-3:13875] Signal code: Non-existant physical address (2) [csclprd3-4-3:13875] Failing at address: 0x2addfef5fe00 [csclprd3-4-3:13883] *** Process received signal *** [csclprd3-4-3:13883] Signal: Bus error (7) [csclprd3-4-3:13883] Signal code: Non-existant physical address (2) [csclprd3-4-3:13883] Failing at address: 0x2b75f9109e80 [csclprd3-4-3:13887] *** Process received signal *** [csclprd3-4-3:13887] Signal: Bus error (7) [csclprd3-4-3:13887] Signal code: Non-existant physical address (2) [csclprd3-4-3:13887] Failing at address: 0x2b58e9c61100 [csclprd3-4-3:13889] *** Process received signal *** [csclprd3-4-3:13889] Signal: Bus error (7) [csclprd3-4-3:13889] Signal code: Non-existant physical address (2) [csclprd3-4-3:13889] Failing at address: 0x2b1367f36c00 [csclprd3-4-3:13895] *** Process received signal *** [csclprd3-4-3:13895] Signal: Bus error (7) [csclprd3-4-3:13895] Signal code: Non-existant physical address (2) [csclprd3-4-3:13895] Failing at address: 0x2b73ff6be080 [csclprd3-4-3:13897] *** Process received signal *** [csclprd3-4-3:13897] Signal: Bus error (7) [csclprd3-4-3:13897] Signal code: Non-existant physical address (2) [csclprd3-4-3:13897] Failing at address: 0x2b945fc62f00 [csclprd3-4-3:13903] *** Process received signal *** [csclprd3-4-3:13903] Signal: Bus error (7) [csclprd3-4-3:13903] Signal code: Non-existant physical address (2) [csclprd3-4-3:13903] Failing at address: 0x2aea87115f38 [csclprd3-4-3:13907] *** Process received signal *** [csclprd3-4-3:13907] Signal: Bus error (7) [csclprd3-4-3:13907] Signal code: Non-existant physical address (2) [csclprd3-4-3:13907] Failing at address: 0x2b372f44adf8 [csclprd3-4-3:13909] *** Process received signal *** [csclprd3-4-3:13909] Signal: Bus error (7) [csclprd3-4-3:13909] Signal code: Non-existant physical address (2) [csclprd3-4-3:13909] Failing at address: 0x2b8f06d6ac18 [csclprd3-4-3:13911] *** Process received signal *** [csclprd3-4-3:13913] *** Process received signal *** [csclprd3-4-3:13913] Signal: Bus error (7) [csclprd3-4-3:13913] Signal code: Non-existant physical address (2) [csclprd3-4-3:13913] Failing at address: 0x2b085ba95b00 [csclprd3-4-3:13921] *** Process received signal *** [csclprd3-4-3:13921] Signal: Bus error (7) [csclprd3-4-3:13921] Signal code: Non-existant physical address (2) [csclprd3-4-3:13921] Failing at address: 0x2b071523d980 [csclprd3-4-3:13923] *** Process received signal *** [csclprd3-4-3:13923] Signal: Bus error (7) [csclprd3-4-3:13923] Signal code: Non-existant physical address (2) [csclprd3-4-3:13923] Failing at address: 0x2b20d4ef0118 [csclprd3-4-3:13850] *** Process received signal *** [csclprd3-4-3:13850] Signal: Bus error (7) [csclprd3-4-3:13850] Signal code: Non-existant physical address (2) [csclprd3-4-3:13850] Failing at address: 0x2ab6b1b7ad58 [csclprd3-4-3:13849] Signal: Bus error (7) [csclprd3-4-3:13849] Signal code: Non-existant physical address (2) [csclprd3-4-3:13849] Failing at address: 0x2acb6366e180 [csclprd3-4-3:13855] *** Process received signal *** [csclprd3-4-3:13855] Signal: Bus error (7) [csclprd3-4-3:13855] Signal code: Non-existant physical address (2) [csclprd3-4-3:13855] Failing at address: 0x2b3e1074be98 [csclprd3-4-3:13861] Signal: Bus error (7) [csclprd3-4-3:13861] Signal code: Non-existant physical address (2) [csclprd3-4-3:13861] Failing at address: 0x2b6c9ddeb000 [csclprd3-4-3:13863] Signal: Bus error (7) [csclprd3-4-3:13863] Signal code: Non-existant physical address (2) [csclprd3-4-3:13863] Failing at address: 0x2b74a9706a00 [csclprd3-4-3:13869] *** Process received signal *** [csclprd3-4-3:13869] Signal: Bus error (7) [csclprd3-4-3:13869] Signal code: Non-existant physical address (2) [csclprd3-4-3:13869] Failing at address: 0x2b4ffebfe258 [csclprd3-4-3:13871] *** Process received signal *** [csclprd3-4-3:13871] Signal: Bus error (7) [csclprd3-4-3:13871] Signal code: Non-existant physical address (2) [csclprd3-4-3:13871] Failing at address: 0x2ac4fd476fd8 [csclprd3-4-3:13873] Signal: Bus error (7) [csclprd3-4-3:13873] Signal code: Non-existant physical address (2) [csclprd3-4-3:13873] Failing at address: 0x2b3d10932b00 [csclprd3-4-3:13877] *** Process received signal *** [csclprd3-4-3:13877] Signal: Bus error (7) [csclprd3-4-3:13877] Signal code: Non-existant physical address (2) [csclprd3-4-3:13877] Failing at address: 0x2b95a73c6cb8 [csclprd3-4-3:13879] *** Process received signal *** [csclprd3-4-3:13879] Signal: Bus error (7) [csclprd3-4-3:13879] Signal code: Non-existant physical address (2) [csclprd3-4-3:13879] Failing at address: 0x2b57908e7078 [csclprd3-4-3:13881] *** Process received signal *** [csclprd3-4-3:13881] Signal: Bus error (7) [csclprd3-4-3:13881] Signal code: Non-existant physical address (2) [csclprd3-4-3:13881] Failing at address: 0x2b4a92947578 [csclprd3-4-3:13885] *** Process received signal *** [csclprd3-4-3:13885] Signal: Bus error (7) [csclprd3-4-3:13885] Signal code: Non-existant physical address (2) [csclprd3-4-3:13885] Failing at address: 0x2ab7867bb1b8 [csclprd3-4-3:13891] *** Process received signal *** [csclprd3-4-3:13891] Signal: Bus error (7) [csclprd3-4-3:13891] Signal code: Non-existant physical address (2) [csclprd3-4-3:13891] Failing at address: 0x2af9a491a2f8 [csclprd3-4-3:13893] *** Process received signal *** [csclprd3-4-3:13893] Signal: Bus error (7) [csclprd3-4-3:13893] Signal code: Non-existant physical address (2) [csclprd3-4-3:13893] Failing at address: 0x2b3531c0d398 [csclprd3-4-3:13899] *** Process received signal *** [csclprd3-4-3:13899] Signal: Bus error (7) [csclprd3-4-3:13899] Signal code: Non-existant physical address (2) [csclprd3-4-3:13899] Failing at address: 0x2b0723e1a6b8 [csclprd3-4-3:13901] *** Process received signal *** [csclprd3-4-3:13901] Signal: Bus error (7) [csclprd3-4-3:13901] Signal code: Non-existant physical address (2) [csclprd3-4-3:13901] Failing at address: 0x2abc946a7618 [csclprd3-4-3:13905] *** Process received signal *** [csclprd3-4-3:13905] Signal: Bus error (7) [csclprd3-4-3:13905] Signal code: Non-existant physical address (2) [csclprd3-4-3:13905] Failing at address: 0x2b44638ec438 [csclprd3-4-3:13911] Signal: Bus error (7) [csclprd3-4-3:13911] Signal code: Non-existant physical address (2) [csclprd3-4-3:13911] Failing at address: 0x2ada368f8980 [csclprd3-4-3:13915] *** Process received signal *** [csclprd3-4-3:13915] Signal: Bus error (7) [csclprd3-4-3:13915] Signal code: Non-existant physical address (2) [csclprd3-4-3:13915] Failing at address: 0x2ae4aa5044d8 [csclprd3-4-3:13917] *** Process received signal *** [csclprd3-4-3:13917] Signal: Bus error (7) [csclprd3-4-3:13917] Signal code: Non-existant physical address (2) [csclprd3-4-3:13917] Failing at address: 0x2b97132517f8 [csclprd3-4-3:13919] *** Process received signal *** [csclprd3-4-3:13919] Signal: Bus error (7) [csclprd3-4-3:13919] Signal code: Non-existant physical address (2) [csclprd3-4-3:13919] Failing at address: 0x2b05c08e1758 [csclprd3-4-3:13853] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b7145c6e500] [csclprd3-4-3:13853] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b714a0e7e8a] [csclprd3-4-3:13853] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b71459a8169] [csclprd3-4-3:13853] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b71459a8270] [csclprd3-4-3:13853] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b7149a4f37e] [csclprd3-4-3:13853] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b714a91a515] [csclprd3-4-3:13853] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b71459c2c36] [csclprd3-4-3:13853] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b71459e1dc0] [csclprd3-4-3:13853] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13853] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7145e9acdd] [csclprd3-4-3:13853] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13853] *** End of error message *** [csclprd3-4-3:13857] Signal: Bus error (7) [csclprd3-4-3:13857] Signal code: Non-existant physical address (2) [csclprd3-4-3:13857] Failing at address: 0x2b3675a93d00 [csclprd3-4-3:13855] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b3dfb60b500] [csclprd3-4-3:13855] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b3dfe9dcf87] [csclprd3-4-3:13855] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b3dfefe6de8] [csclprd3-4-3:13855] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b3dfb34506d] [csclprd3-4-3:13855] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b3dfb345270] [csclprd3-4-3:13855] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b3dff3ec37e] [csclprd3-4-3:13855] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b3e04428515] [csclprd3-4-3:13855] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b3dfb35fc36] [csclprd3-4-3:13855] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b3dfb37edc0] [csclprd3-4-3:13855] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13855] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b3dfb837cdd] [csclprd3-4-3:13855] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13855] *** End of error message *** [csclprd3-4-3:13867] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b7f07ed9500] [csclprd3-4-3:13867] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b7f10495e8a] [csclprd3-4-3:13867] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b7f07c13169] [csclprd3-4-3:13867] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b7f07c13270] [csclprd3-4-3:13867] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b7f0bcba37e] [csclprd3-4-3:13867] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b7f10cc8515] [csclprd3-4-3:13867] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b7f07c2dc36] [csclprd3-4-3:13867] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b7f07c4cdc0] [csclprd3-4-3:13867] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13867] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7f08105cdd] [csclprd3-4-3:13867] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13867] *** End of error message *** [csclprd3-4-3:13877] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b9592342500] [csclprd3-4-3:13877] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b9595713f87] [csclprd3-4-3:13877] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b9595d1dde8] [csclprd3-4-3:13877] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b959207c06d] [csclprd3-4-3:13877] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b959207c270] [csclprd3-4-3:13877] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b959612337e] [csclprd3-4-3:13877] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b9596fee515] [csclprd3-4-3:13877] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b9592096c36] [csclprd3-4-3:13877] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b95920b5dc0] [csclprd3-4-3:13877] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13877] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b959256ecdd] [csclprd3-4-3:13877] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13877] *** End of error message *** [csclprd3-4-3:13879] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b577b3e2500] [csclprd3-4-3:13879] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b577e59cf87] [csclprd3-4-3:13879] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b577eba6de8] [csclprd3-4-3:13879] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b577b11c06d] [csclprd3-4-3:13879] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b577b11c270] [csclprd3-4-3:13879] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b577efac37e] [csclprd3-4-3:13879] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b5784004515] [csclprd3-4-3:13879] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b577b136c36] [csclprd3-4-3:13879] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b577b155dc0] [csclprd3-4-3:13879] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13879] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b577b60ecdd] [csclprd3-4-3:13879] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13879] *** End of error message *** [csclprd3-4-3:13875] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2addea14e500] [csclprd3-4-3:13875] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2addee5bee8a] [csclprd3-4-3:13875] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2adde9e88169] [csclprd3-4-3:13875] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2adde9e88270] [csclprd3-4-3:13875] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2addedf2637e] [csclprd3-4-3:13875] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2addeedf1515] [csclprd3-4-3:13875] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2adde9ea2c36] [csclprd3-4-3:13875] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2adde9ec1dc0] [csclprd3-4-3:13875] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13875] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2addea37acdd] [csclprd3-4-3:13875] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13875] *** End of error message *** [csclprd3-4-3:13883] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b75e4413500] [csclprd3-4-3:13883] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b75ec89ce8a] [csclprd3-4-3:13883] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b75e414d169] [csclprd3-4-3:13883] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b75e414d270] [csclprd3-4-3:13883] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b75ec20437e] [csclprd3-4-3:13883] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b75ed0cf515] [csclprd3-4-3:13883] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b75e4167c36] [csclprd3-4-3:13883] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b75e4186dc0] [csclprd3-4-3:13883] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13883] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b75e463fcdd] [csclprd3-4-3:13883] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13883] *** End of error message *** [csclprd3-4-3:13889] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1353297500] [csclprd3-4-3:13889] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b1357710e8a] [csclprd3-4-3:13889] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b1352fd1169] [csclprd3-4-3:13889] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b1352fd1270] [csclprd3-4-3:13889] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b135707837e] [csclprd3-4-3:13889] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b135c004515] [csclprd3-4-3:13889] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b1352febc36] [csclprd3-4-3:13889] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b135300adc0] [csclprd3-4-3:13889] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13889] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b13534c3cdd] [csclprd3-4-3:13889] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13889] *** End of error message *** [csclprd3-4-3:13895] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b73ea79a500] [csclprd3-4-3:13895] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b73eec13e8a] [csclprd3-4-3:13895] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b73ea4d4169] [csclprd3-4-3:13895] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b73ea4d4270] [csclprd3-4-3:13895] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b73ee57b37e] [csclprd3-4-3:13895] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b73ef446515] [csclprd3-4-3:13895] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b73ea4eec36] [csclprd3-4-3:13895] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b73ea50ddc0] [csclprd3-4-3:13895] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13895] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b73ea9c6cdd] [csclprd3-4-3:13895] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13895] *** End of error message *** [csclprd3-4-3:13897] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b944add9500] [csclprd3-4-3:13897] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b944f252e8a] [csclprd3-4-3:13897] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b944ab13169] [csclprd3-4-3:13897] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b944ab13270] [csclprd3-4-3:13897] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b944ebba37e] [csclprd3-4-3:13897] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b944fa85515] [csclprd3-4-3:13897] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b944ab2dc36] [csclprd3-4-3:13897] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b944ab4cdc0] [csclprd3-4-3:13897] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13897] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b944b005cdd] [csclprd3-4-3:13897] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13897] *** End of error message *** [csclprd3-4-3:13907] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b371a2f2500] [csclprd3-4-3:13907] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b371d6c3f87] [csclprd3-4-3:13907] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b371dccdde8] [csclprd3-4-3:13907] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b371a02c06d] [csclprd3-4-3:13907] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b371a02c270] [csclprd3-4-3:13907] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b371e0d337e] [csclprd3-4-3:13907] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b371ef9e515] [csclprd3-4-3:13907] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b371a046c36] [csclprd3-4-3:13907] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b371a065dc0] [csclprd3-4-3:13907] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13907] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b371a51ecdd] [csclprd3-4-3:13907] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13907] *** End of error message *** [csclprd3-4-3:13909] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b8ef1c7c500] [csclprd3-4-3:13909] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b8ef504df87] [csclprd3-4-3:13909] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b8ef5657de8] [csclprd3-4-3:13909] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b8ef19b606d] [csclprd3-4-3:13909] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b8ef19b6270] [csclprd3-4-3:13909] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b8ef5a5d37e] [csclprd3-4-3:13909] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b8ef6928515] [csclprd3-4-3:13909] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b8ef19d0c36] [csclprd3-4-3:13909] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b8ef19efdc0] [csclprd3-4-3:13909] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13909] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b8ef1ea8cdd] [csclprd3-4-3:13909] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13909] *** End of error message *** [csclprd3-4-3:13923] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b20bfcc4500] [csclprd3-4-3:13923] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b20c3095f87] [csclprd3-4-3:13923] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b20c369fde8] [csclprd3-4-3:13923] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b20bf9fe06d] [csclprd3-4-3:13923] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b20bf9fe270] [csclprd3-4-3:13923] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b20c3aa537e] [csclprd3-4-3:13923] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b20c8ac4515] [csclprd3-4-3:13923] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b20bfa18c36] [csclprd3-4-3:13923] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b20bfa37dc0] [csclprd3-4-3:13923] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13923] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b20bfef0cdd] [csclprd3-4-3:13923] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13923] *** End of error message *** [csclprd3-4-3:13850] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab69caf1500] [csclprd3-4-3:13850] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab6a4000f87] [csclprd3-4-3:13850] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab6a460ade8] [csclprd3-4-3:13850] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab69c82b06d] [csclprd3-4-3:13850] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab69c82b270] [csclprd3-4-3:13850] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab6a4a1037e] [csclprd3-4-3:13850] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab6a58db515] [csclprd3-4-3:13850] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab69c845c36] [csclprd3-4-3:13850] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab69c864dc0] [csclprd3-4-3:13850] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13850] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab69cd1dcdd] [csclprd3-4-3:13850] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13850] *** End of error message *** [csclprd3-4-3:13851] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b06c1f30500] [csclprd3-4-3:13851] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b06c63a9e8a] [csclprd3-4-3:13851] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b06c1c6a169] [csclprd3-4-3:13851] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b06c1c6a270] [csclprd3-4-3:13851] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b06c5d1137e] [csclprd3-4-3:13851] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b06c6bdc515] [csclprd3-4-3:13851] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b06c1c84c36] [csclprd3-4-3:13851] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b06c1ca3dc0] [csclprd3-4-3:13851] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13851] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b06c215ccdd] [csclprd3-4-3:13851] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13851] *** End of error message *** [csclprd3-4-3:13915] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ae4951a2500] [csclprd3-4-3:13915] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ae498573f87] [csclprd3-4-3:13915] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ae498b7dde8] [csclprd3-4-3:13915] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ae494edc06d] [csclprd3-4-3:13915] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ae494edc270] [csclprd3-4-3:13915] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ae498f8337e] [csclprd3-4-3:13915] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ae499e4e515] [csclprd3-4-3:13915] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ae494ef6c36] [csclprd3-4-3:13915] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ae494f15dc0] [csclprd3-4-3:13915] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13915] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ae4953cecdd] [csclprd3-4-3:13915] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13915] *** End of error message *** [csclprd3-4-3:13913] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b0846e14500] [csclprd3-4-3:13913] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b084b28de8a] [csclprd3-4-3:13913] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b0846b4e169] [csclprd3-4-3:13913] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b0846b4e270] [csclprd3-4-3:13913] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b084abf537e] [csclprd3-4-3:13913] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b084bac0515] [csclprd3-4-3:13913] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b0846b68c36] [csclprd3-4-3:13913] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b0846b87dc0] [csclprd3-4-3:13913] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13913] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0847040cdd] [csclprd3-4-3:13913] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13913] *** End of error message *** [csclprd3-4-3:13921] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b0700719500] [csclprd3-4-3:13921] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b0708ca2e8a] [csclprd3-4-3:13921] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b0700453169] [csclprd3-4-3:13921] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b0700453270] [csclprd3-4-3:13921] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b070860a37e] [csclprd3-4-3:13921] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b07094d5515] [csclprd3-4-3:13921] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b070046dc36] [csclprd3-4-3:13921] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b070048cdc0] [csclprd3-4-3:13921] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13921] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0700945cdd] [csclprd3-4-3:13921] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13921] *** End of error message *** [csclprd3-4-3:13849] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2acb582b5500] [csclprd3-4-3:13849] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2acb6089ce8a] [csclprd3-4-3:13849] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2acb57fef169] [csclprd3-4-3:13849] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2acb57fef270] [csclprd3-4-3:13849] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2acb6020437e] [csclprd3-4-3:13849] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2acb610cf515] [csclprd3-4-3:13849] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2acb58009c36] [csclprd3-4-3:13849] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2acb58028dc0] [csclprd3-4-3:13849] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13849] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2acb584e1cdd] [csclprd3-4-3:13849] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13849] *** End of error message *** [csclprd3-4-3:13857] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b3660e13500] [csclprd3-4-3:13857] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b36692aae8a] [csclprd3-4-3:13857] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b3660b4d169] [csclprd3-4-3:13857] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b3660b4d270] [csclprd3-4-3:13857] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b3668c1237e] [csclprd3-4-3:13857] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b3669add515] [csclprd3-4-3:13857] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b3660b67c36] [csclprd3-4-3:13857] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b3660b86dc0] [csclprd3-4-3:13857] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13857] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b366103fcdd] [csclprd3-4-3:13857] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13857] *** End of error message *** [csclprd3-4-3:13863] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b7494b95500] [csclprd3-4-3:13863] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b749d0a8e8a] [csclprd3-4-3:13863] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b74948cf169] [csclprd3-4-3:13863] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b74948cf270] [csclprd3-4-3:13863] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b749ca1037e] [csclprd3-4-3:13863] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b749d8db515] [csclprd3-4-3:13863] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b74948e9c36] [csclprd3-4-3:13863] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b7494908dc0] [csclprd3-4-3:13863] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13863] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7494dc1cdd] [csclprd3-4-3:13863] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13863] *** End of error message *** [csclprd3-4-3:13869] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b4fe9967500] [csclprd3-4-3:13869] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b4fecd2ff87] [csclprd3-4-3:13869] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b4fed339de8] [csclprd3-4-3:13869] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b4fe96a106d] [csclprd3-4-3:13869] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b4fe96a1270] [csclprd3-4-3:13869] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b4fed73f37e] [csclprd3-4-3:13869] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b4fee60a515] [csclprd3-4-3:13869] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b4fe96bbc36] [csclprd3-4-3:13869] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b4fe96dadc0] [csclprd3-4-3:13869] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13869] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4fe9b93cdd] [csclprd3-4-3:13869] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13869] *** End of error message *** [csclprd3-4-3:13871] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ac4e83ac500] [csclprd3-4-3:13871] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ac4eb77df87] [csclprd3-4-3:13871] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ac4ebd87de8] [csclprd3-4-3:13871] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ac4e80e606d] [csclprd3-4-3:13871] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ac4e80e6270] [csclprd3-4-3:13871] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ac4f020437e] [csclprd3-4-3:13871] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ac4f10cf515] [csclprd3-4-3:13871] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ac4e8100c36] [csclprd3-4-3:13871] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ac4e811fdc0] [csclprd3-4-3:13871] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13871] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac4e85d8cdd] [csclprd3-4-3:13871] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13871] *** End of error message *** [csclprd3-4-3:13881] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b4a7d506500] [csclprd3-4-3:13881] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b4a808d7f87] [csclprd3-4-3:13881] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b4a80ee1de8] [csclprd3-4-3:13881] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b4a7d24006d] [csclprd3-4-3:13881] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b4a7d240270] [csclprd3-4-3:13881] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b4a812e737e] [csclprd3-4-3:13881] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b4a821b2515] [csclprd3-4-3:13881] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b4a7d25ac36] [csclprd3-4-3:13881] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b4a7d279dc0] [csclprd3-4-3:13881] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13881] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4a7d732cdd] [csclprd3-4-3:13881] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13881] *** End of error message *** [csclprd3-4-3:13887] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b58d4e25500] [csclprd3-4-3:13887] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b58dd2aae8a] [csclprd3-4-3:13887] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b58d4b5f169] [csclprd3-4-3:13887] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b58d4b5f270] [csclprd3-4-3:13887] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b58dcc1237e] [csclprd3-4-3:13887] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b58ddadd515] [csclprd3-4-3:13887] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b58d4b79c36] [csclprd3-4-3:13887] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b58d4b98dc0] [csclprd3-4-3:13887] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13887] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b58d5051cdd] [csclprd3-4-3:13887] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13887] *** End of error message *** [csclprd3-4-3:13885] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab7712e9500] [csclprd3-4-3:13885] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab7746baf87] [csclprd3-4-3:13885] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab774cc4de8] [csclprd3-4-3:13885] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab77102306d] [csclprd3-4-3:13885] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab771023270] [csclprd3-4-3:13885] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab7750ca37e] [csclprd3-4-3:13885] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab775f95515] [csclprd3-4-3:13885] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab77103dc36] [csclprd3-4-3:13885] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab77105cdc0] [csclprd3-4-3:13885] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13885] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab771515cdd] [csclprd3-4-3:13885] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13885] *** End of error message *** [csclprd3-4-3:13891] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2af98f779500] [csclprd3-4-3:13891] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2af992b41f87] [csclprd3-4-3:13891] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2af99314bde8] [csclprd3-4-3:13891] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2af98f4b306d] [csclprd3-4-3:13891] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2af98f4b3270] [csclprd3-4-3:13891] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2af99355137e] [csclprd3-4-3:13891] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2af998428515] [csclprd3-4-3:13891] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2af98f4cdc36] [csclprd3-4-3:13891] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2af98f4ecdc0] [csclprd3-4-3:13891] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13891] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2af98f9a5cdd] [csclprd3-4-3:13891] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13891] *** End of error message *** [csclprd3-4-3:13893] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b351ca0f500] [csclprd3-4-3:13893] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b351fde0f87] [csclprd3-4-3:13893] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b3524408de8] [csclprd3-4-3:13893] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b351c74906d] [csclprd3-4-3:13893] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b351c749270] [csclprd3-4-3:13893] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b352480e37e] [csclprd3-4-3:13893] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b35256d9515] [csclprd3-4-3:13893] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b351c763c36] [csclprd3-4-3:13893] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b351c782dc0] [csclprd3-4-3:13893] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13893] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b351cc3bcdd] [csclprd3-4-3:13893] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13893] *** End of error message *** [csclprd3-4-3:13903] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2aea72096500] [csclprd3-4-3:13903] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2aea75250f87] [csclprd3-4-3:13903] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2aea7585ade8] [csclprd3-4-3:13903] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2aea71dd006d] [csclprd3-4-3:13903] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2aea71dd0270] [csclprd3-4-3:13903] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2aea75c6037e] [csclprd3-4-3:13903] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2aea76b2b515] [csclprd3-4-3:13903] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2aea71deac36] [csclprd3-4-3:13903] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2aea71e09dc0] [csclprd3-4-3:13903] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13903] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2aea722c2cdd] [csclprd3-4-3:13903] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13903] *** End of error message *** [csclprd3-4-3:13905] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b444e67c500] [csclprd3-4-3:13905] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b4451a4df87] [csclprd3-4-3:13905] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b4452057de8] [csclprd3-4-3:13905] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b444e3b606d] [csclprd3-4-3:13905] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b444e3b6270] [csclprd3-4-3:13905] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b445245d37e] [csclprd3-4-3:13905] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b4453328515] [csclprd3-4-3:13905] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b444e3d0c36] [csclprd3-4-3:13905] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b444e3efdc0] [csclprd3-4-3:13905] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13905] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b444e8a8cdd] [csclprd3-4-3:13905] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13905] *** End of error message *** [csclprd3-4-3:13919] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b05ab404500] [csclprd3-4-3:13919] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b05ae7d5f87] [csclprd3-4-3:13919] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b05aeddfde8] [csclprd3-4-3:13919] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b05ab13e06d] [csclprd3-4-3:13919] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b05ab13e270] [csclprd3-4-3:13919] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b05af1e537e] [csclprd3-4-3:13919] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b05b4221515] [csclprd3-4-3:13919] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b05ab158c36] [csclprd3-4-3:13919] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b05ab177dc0] [csclprd3-4-3:13919] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13919] [10] [csclprd3-4-3:13861] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b6c88e50500] [csclprd3-4-3:13861] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b6c914b8e8a] [csclprd3-4-3:13861] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b6c88b8a169] [csclprd3-4-3:13861] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b6c88b8a270] [csclprd3-4-3:13861] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b6c90e2037e] [csclprd3-4-3:13861] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b6c91ceb515] [csclprd3-4-3:13861] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b6c88ba4c36] [csclprd3-4-3:13861] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b6c88bc3dc0] [csclprd3-4-3:13861] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13861] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6c8907ccdd] [csclprd3-4-3:13861] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13861] *** End of error message *** [csclprd3-4-3:13873] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b3cfbd97500] [csclprd3-4-3:13873] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b3d04291e8a] [csclprd3-4-3:13873] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b3cfbad1169] [csclprd3-4-3:13873] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b3cfbad1270] [csclprd3-4-3:13873] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b3cffb7837e] [csclprd3-4-3:13873] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b3d04ac4515] [csclprd3-4-3:13873] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b3cfbaebc36] [csclprd3-4-3:13873] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b3cfbb0adc0] [csclprd3-4-3:13873] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13873] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b3cfbfc3cdd] [csclprd3-4-3:13873] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13873] *** End of error message *** [csclprd3-4-3:13899] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b070ea4b500] [csclprd3-4-3:13899] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b0711e13f87] [csclprd3-4-3:13899] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b071241dde8] [csclprd3-4-3:13899] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b070e78506d] [csclprd3-4-3:13899] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b070e785270] [csclprd3-4-3:13899] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b071282337e] [csclprd3-4-3:13899] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b07136ee515] [csclprd3-4-3:13899] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b070e79fc36] [csclprd3-4-3:13899] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b070e7bedc0] [csclprd3-4-3:13899] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13899] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b070ec77cdd] [csclprd3-4-3:13899] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13899] *** End of error message *** [csclprd3-4-3:13901] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2abc7f351500] [csclprd3-4-3:13901] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2abc8250bf87] [csclprd3-4-3:13901] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2abc82b15de8] [csclprd3-4-3:13901] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2abc7f08b06d] [csclprd3-4-3:13901] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2abc7f08b270] [csclprd3-4-3:13901] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2abc82f1b37e] [csclprd3-4-3:13901] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2abc83de6515] [csclprd3-4-3:13901] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2abc7f0a5c36] [csclprd3-4-3:13901] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2abc7f0c4dc0] [csclprd3-4-3:13901] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13901] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2abc7f57dcdd] [csclprd3-4-3:13901] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13901] *** End of error message *** [csclprd3-4-3:13911] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ada21ccf500] [csclprd3-4-3:13911] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2ada26148e8a] [csclprd3-4-3:13911] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2ada21a09169] [csclprd3-4-3:13911] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ada21a09270] [csclprd3-4-3:13911] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ada25ab037e] [csclprd3-4-3:13911] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ada2697b515] [csclprd3-4-3:13911] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ada21a23c36] [csclprd3-4-3:13911] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ada21a42dc0] [csclprd3-4-3:13911] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13911] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ada21efbcdd] [csclprd3-4-3:13911] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13911] *** End of error message *** [csclprd3-4-3:13917] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b96fdd1a500] [csclprd3-4-3:13917] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b97010ebf87] [csclprd3-4-3:13917] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b97016f5de8] [csclprd3-4-3:13917] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b96fda5406d] [csclprd3-4-3:13917] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b96fda54270] [csclprd3-4-3:13917] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b9701afb37e] [csclprd3-4-3:13917] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b97029c6515] [csclprd3-4-3:13917] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b96fda6ec36] [csclprd3-4-3:13917] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b96fda8ddc0] [csclprd3-4-3:13917] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13917] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b96fdf46cdd] [csclprd3-4-3:13917] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13917] *** End of error message *** /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b05ab630cdd] [csclprd3-4-3:13919] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13919] *** End of error message *** [csclprd3-4-3:13865] Signal: Bus error (7) [csclprd3-4-3:13865] Signal code: Non-existant physical address (2) [csclprd3-4-3:13865] Failing at address: 0x2b5d4c194c00 [csclprd3-4-3:13859] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b9a1c96c500] [csclprd3-4-3:13859] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b9a24ea6e8a] [csclprd3-4-3:13859] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b9a1c6a6169] [csclprd3-4-3:13859] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b9a1c6a6270] [csclprd3-4-3:13859] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b9a2480e37e] [csclprd3-4-3:13859] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b9a256d9515] [csclprd3-4-3:13859] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b9a1c6c0c36] [csclprd3-4-3:13859] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b9a1c6dfdc0] [csclprd3-4-3:13859] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13859] [ 9] [csclprd3-4-3:13865] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b5d37385500] [csclprd3-4-3:13865] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b5d3b7fee8a] [csclprd3-4-3:13865] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b5d370bf169] [csclprd3-4-3:13865] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b5d370bf270] [csclprd3-4-3:13865] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b5d3b16637e] [csclprd3-4-3:13865] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b5d40221515] [csclprd3-4-3:13865] [ 6] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b9a1cb98cdd] [csclprd3-4-3:13859] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13859] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b5d370d9c36] [csclprd3-4-3:13865] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b5d370f8dc0] [csclprd3-4-3:13865] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-4-3:13865] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b5d375b1cdd] [csclprd3-4-3:13865] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-4-3:13865] *** End of error message *** [csclprd3-0-0:05904] *** Process received signal *** [csclprd3-0-0:05904] Signal: Bus error (7) [csclprd3-0-0:05904] Signal code: Non-existant physical address (2) [csclprd3-0-0:05904] Failing at address: 0x2afe58acfa00 [csclprd3-0-0:05904] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2afe45e44500] [csclprd3-0-0:05904] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2afe4a2bde8a] [csclprd3-0-0:05904] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2afe45b7e169] [csclprd3-0-0:05904] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2afe45b7e270] [csclprd3-0-0:05904] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2afe49c2537e] [csclprd3-0-0:05904] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2afe4aaf0515] [csclprd3-0-0:05904] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2afe45b98c36] [csclprd3-0-0:05904] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2afe45bb7dc0] [csclprd3-0-0:05904] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05904] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2afe46070cdd] [csclprd3-0-0:05904] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05904] *** End of error message *** [csclprd3-0-0:05898] *** Process received signal *** [csclprd3-0-0:05898] Signal: Bus error (7) [csclprd3-0-0:05898] Signal code: Non-existant physical address (2) [csclprd3-0-0:05898] Failing at address: 0x2ad4210f3f18 [csclprd3-0-0:05898] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ad40e3dd500] [csclprd3-0-0:05898] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ad4117aef87] [csclprd3-0-0:05898] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ad411db8de8] [csclprd3-0-0:05898] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ad40e11706d] [csclprd3-0-0:05898] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ad40e117270] [csclprd3-0-0:05898] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ad4121be37e] [csclprd3-0-0:05898] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ad413089515] [csclprd3-0-0:05898] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ad40e131c36] [csclprd3-0-0:05898] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ad40e150dc0] [csclprd3-0-0:05898] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05898] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad40e609cdd] [csclprd3-0-0:05898] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05898] *** End of error message *** [csclprd3-0-0:05900] *** Process received signal *** [csclprd3-0-0:05900] Signal: Bus error (7) [csclprd3-0-0:05900] Signal code: Non-existant physical address (2) [csclprd3-0-0:05900] Failing at address: 0x2add05bb2fb8 [csclprd3-0-0:05900] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2adcf2f3d500] [csclprd3-0-0:05900] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2adcf6305f87] [csclprd3-0-0:05900] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2adcf690fde8] [csclprd3-0-0:05900] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2adcf2c7706d] [csclprd3-0-0:05900] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2adcf2c77270] [csclprd3-0-0:05900] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2adcf6d1537e] [csclprd3-0-0:05900] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2adcf7be0515] [csclprd3-0-0:05900] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2adcf2c91c36] [csclprd3-0-0:05900] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2adcf2cb0dc0] [csclprd3-0-0:05900] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05900] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2adcf3169cdd] [csclprd3-0-0:05900] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05900] *** End of error message *** [csclprd3-0-0:05910] *** Process received signal *** [csclprd3-0-0:05910] Signal: Bus error (7) [csclprd3-0-0:05910] Signal code: Non-existant physical address (2) [csclprd3-0-0:05910] Failing at address: 0x2aed52013058 [csclprd3-0-0:05910] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2aed3f249500] [csclprd3-0-0:05910] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2aed4261af87] [csclprd3-0-0:05910] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2aed42c24de8] [csclprd3-0-0:05910] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2aed3ef8306d] [csclprd3-0-0:05910] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2aed3ef83270] [csclprd3-0-0:05910] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2aed4302a37e] [csclprd3-0-0:05910] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2aed48004515] [csclprd3-0-0:05910] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2aed3ef9dc36] [csclprd3-0-0:05910] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2aed3efbcdc0] [csclprd3-0-0:05910] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05910] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2aed3f475cdd] [csclprd3-0-0:05910] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05910] *** End of error message *** [csclprd3-0-0:05892] *** Process received signal *** [csclprd3-0-0:05892] Signal: Bus error (7) [csclprd3-0-0:05892] Signal code: Non-existant physical address (2) [csclprd3-0-0:05892] Failing at address: 0x2ba88839e0f8 [csclprd3-0-0:05892] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ba875696500] [csclprd3-0-0:05892] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ba878a67f87] [csclprd3-0-0:05892] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ba879071de8] [csclprd3-0-0:05892] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ba8753d006d] [csclprd3-0-0:05892] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ba8753d0270] [csclprd3-0-0:05892] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ba87947737e] [csclprd3-0-0:05892] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ba87a342515] [csclprd3-0-0:05892] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ba8753eac36] [csclprd3-0-0:05892] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ba875409dc0] [csclprd3-0-0:05892] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05892] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ba8758c2cdd] [csclprd3-0-0:05892] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05892] *** End of error message *** [csclprd3-0-0:05940] *** Process received signal *** [csclprd3-0-0:05940] Signal: Bus error (7) [csclprd3-0-0:05940] Signal code: Non-existant physical address (2) [csclprd3-0-0:05940] Failing at address: 0x2b1bd22b4198 [csclprd3-0-0:05940] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1bbf3e6500] [csclprd3-0-0:05940] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b1bc27b7f87] [csclprd3-0-0:05940] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b1bc2dc1de8] [csclprd3-0-0:05940] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b1bbf12006d] [csclprd3-0-0:05940] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b1bbf120270] [csclprd3-0-0:05940] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b1bc31c737e] [csclprd3-0-0:05940] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b1bc8221515] [csclprd3-0-0:05940] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b1bbf13ac36] [csclprd3-0-0:05940] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b1bbf159dc0] [csclprd3-0-0:05940] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05940] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1bbf612cdd] [csclprd3-0-0:05940] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05940] *** End of error message *** [csclprd3-0-0:05917] *** Process received signal *** [csclprd3-0-0:05917] Signal: Bus error (7) [csclprd3-0-0:05917] Signal code: Non-existant physical address (2) [csclprd3-0-0:05917] Failing at address: 0x2b1e35ec2238 [csclprd3-0-0:05917] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1e23079500] [csclprd3-0-0:05917] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b1e2644af87] [csclprd3-0-0:05917] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b1e26a54de8] [csclprd3-0-0:05917] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b1e22db306d] [csclprd3-0-0:05917] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b1e22db3270] [csclprd3-0-0:05917] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b1e26e5a37e] [csclprd3-0-0:05917] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b1e27d25515] [csclprd3-0-0:05917] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b1e22dcdc36] [csclprd3-0-0:05917] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b1e22decdc0] [csclprd3-0-0:05917] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05917] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1e232a5cdd] [csclprd3-0-0:05917] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05917] *** End of error message *** [csclprd3-0-0:05928] *** Process received signal *** [csclprd3-0-0:05928] Signal: Bus error (7) [csclprd3-0-0:05928] Signal code: Non-existant physical address (2) [csclprd3-0-0:05928] Failing at address: 0x2b1bc9cfd2d8 [csclprd3-0-15:16525] *** Process received signal *** [csclprd3-0-15:16525] Signal: Bus error (7) [csclprd3-0-15:16525] Signal code: Non-existant physical address (2) [csclprd3-0-15:16525] Failing at address: 0x2ab410197080 [csclprd3-0-0:05928] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1bb6f16500] [csclprd3-0-0:05928] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b1bba2e7f87] [csclprd3-0-0:05928] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b1bba8f1de8] [csclprd3-0-0:05928] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b1bb6c5006d] [csclprd3-0-0:05928] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b1bb6c50270] [csclprd3-0-0:05928] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b1bbacf737e] [csclprd3-0-0:05928] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b1bbbbc2515] [csclprd3-0-0:05928] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b1bb6c6ac36] [csclprd3-0-0:05928] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b1bb6c89dc0] [csclprd3-0-0:05928] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05928] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1bb7142cdd] [csclprd3-0-0:05928] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05928] *** End of error message *** [csclprd3-0-0:05908] *** Process received signal *** [csclprd3-0-0:05908] Signal: Bus error (7) [csclprd3-0-0:05908] Signal code: Non-existant physical address (2) [csclprd3-0-0:05908] Failing at address: 0x2b670d8f5378 [csclprd3-0-15:16527] *** Process received signal *** [csclprd3-0-15:16527] Signal: Bus error (7) [csclprd3-0-15:16527] Signal code: Non-existant physical address (2) [csclprd3-0-15:16527] Failing at address: 0x2b2f132c5038 [csclprd3-0-0:05908] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b66fa9b8500] [csclprd3-0-0:05908] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b66fdd89f87] [csclprd3-0-0:05908] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b66fe393de8] [csclprd3-0-0:05908] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b66fa6f206d] [csclprd3-0-0:05908] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b66fa6f2270] [csclprd3-0-0:05908] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b66fe79937e] [csclprd3-0-0:05908] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b66ff664515] [csclprd3-0-0:05908] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b66fa70cc36] [csclprd3-0-0:05908] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b66fa72bdc0] [csclprd3-0-0:05908] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05908] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b66fabe4cdd] [csclprd3-0-0:05908] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05908] *** End of error message *** [csclprd3-0-15:16537] *** Process received signal *** [csclprd3-0-15:16543] *** Process received signal *** [csclprd3-0-15:16543] Signal: Bus error (7) [csclprd3-0-15:16543] Signal code: Non-existant physical address (2) [csclprd3-0-15:16543] Failing at address: 0x2ab92bd340d8 [csclprd3-0-0:05937] *** Process received signal *** [csclprd3-0-0:05937] Signal: Bus error (7) [csclprd3-0-0:05937] Signal code: Non-existant physical address (2) [csclprd3-0-0:05937] Failing at address: 0x2b89da5c3418 [csclprd3-0-15:16549] *** Process received signal *** [csclprd3-0-0:05937] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b89c7669500] [csclprd3-0-0:05937] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b89caa3af87] [csclprd3-0-0:05937] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b89cb044de8] [csclprd3-0-0:05937] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b89c73a306d] [csclprd3-0-0:05937] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b89c73a3270] [csclprd3-0-0:05937] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b89cb44a37e] [csclprd3-0-0:05937] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b89d0428515] [csclprd3-0-0:05937] [ 7] [csclprd3-0-15:16481] *** Process received signal *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b89c73bdc36] [csclprd3-0-0:05937] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b89c73dcdc0] [csclprd3-0-0:05937] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05937] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b89c7895cdd] [csclprd3-0-0:05937] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05937] *** End of error message *** [csclprd3-0-15:16482] *** Process received signal *** [csclprd3-0-15:16482] Signal: Bus error (7) [csclprd3-0-15:16482] Signal code: Non-existant physical address (2) [csclprd3-0-15:16482] Failing at address: 0x2b07cc4a7380 [csclprd3-0-0:05926] *** Process received signal *** [csclprd3-0-0:05926] Signal: Bus error (7) [csclprd3-0-0:05926] Signal code: Non-existant physical address (2) [csclprd3-0-0:05926] Failing at address: 0x2b98623fe4b8 [csclprd3-0-15:16485] *** Process received signal *** [csclprd3-0-0:05926] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b984f3ab500] [csclprd3-0-0:05926] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b985277cf87] [csclprd3-0-0:05926] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b9852d86de8] [csclprd3-0-0:05926] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b984f0e506d] [csclprd3-0-0:05926] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b984f0e5270] [csclprd3-0-0:05926] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b985318c37e] [csclprd3-0-0:05926] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b9858221515] [csclprd3-0-0:05926] [ 7] [csclprd3-0-15:16489] *** Process received signal *** [csclprd3-0-15:16489] Signal: Bus error (7) [csclprd3-0-15:16489] Signal code: Non-existant physical address (2) [csclprd3-0-15:16489] Failing at address: 0x2b9f0825e300 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b984f0ffc36] [csclprd3-0-0:05926] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b984f11edc0] [csclprd3-0-0:05926] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05926] [10] [csclprd3-0-15:16493] *** Process received signal *** /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b984f5d7cdd] [csclprd3-0-0:05926] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05926] *** End of error message *** [csclprd3-0-15:16495] *** Process received signal *** [csclprd3-0-15:16495] Signal: Bus error (7) [csclprd3-0-15:16495] Signal code: Non-existant physical address (2) [csclprd3-0-15:16495] Failing at address: 0x2ac585dc8000 [csclprd3-0-0:05935] *** Process received signal *** [csclprd3-0-0:05935] Signal: Bus error (7) [csclprd3-0-0:05935] Signal code: Non-existant physical address (2) [csclprd3-0-0:05935] Failing at address: 0x2b9c919bb558 [csclprd3-0-15:16499] *** Process received signal *** [csclprd3-0-15:16499] Signal: Bus error (7) [csclprd3-0-15:16499] Signal code: Non-existant physical address (2) [csclprd3-0-15:16499] Failing at address: 0x2b8d4fc47f80 [csclprd3-0-0:05935] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b9c7ea82500] [csclprd3-0-0:05935] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b9c81e53f87] [csclprd3-0-0:05935] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b9c8245dde8] [csclprd3-0-0:05935] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b9c7e7bc06d] [csclprd3-0-0:05935] [csclprd3-0-15:16501] *** Process received signal *** [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b9c7e7bc270] [csclprd3-0-0:05935] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b9c8286337e] [csclprd3-0-0:05935] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b9c8372e515] [csclprd3-0-0:05935] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b9c7e7d6c36] [csclprd3-0-0:05935] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b9c7e7f5dc0] [csclprd3-0-0:05935] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05935] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b9c7ecaecdd] [csclprd3-0-0:05935] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05935] *** End of error message *** [csclprd3-0-0:05919] *** Process received signal *** [csclprd3-0-0:05919] Signal: Bus error (7) [csclprd3-0-0:05919] Signal code: Non-existant physical address (2) [csclprd3-0-0:05919] Failing at address: 0x2b16fbb3c5f8 [csclprd3-0-15:16495] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ac570ac8500] [csclprd3-0-15:16495] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2ac5790a8e8a] [csclprd3-0-15:16495] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2ac570802169] [csclprd3-0-15:16495] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ac570802270] [csclprd3-0-15:16495] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ac578a1037e] [csclprd3-0-15:16495] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ac5798db515] [csclprd3-0-15:16495] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ac57081cc36] [csclprd3-0-15:16495] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ac57083bdc0] [csclprd3-0-15:16495] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16495] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac570cf4cdd] [csclprd3-0-15:16495] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16495] *** End of error message *** [csclprd3-0-0:05919] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b16e8ad8500] [csclprd3-0-0:05919] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b16f0000f87] [csclprd3-0-0:05919] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b16f060ade8] [csclprd3-0-0:05919] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b16e881206d] [csclprd3-0-0:05919] [ 4] [csclprd3-0-15:16499] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b8d3a96a500] [csclprd3-0-15:16499] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b8d3ede3e8a] [csclprd3-0-15:16499] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b8d3a6a4169] [csclprd3-0-15:16499] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b8d3a6a4270] [csclprd3-0-15:16499] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b8d3e74b37e] [csclprd3-0-15:16499] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b8d3f616515] [csclprd3-0-15:16499] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b8d3a6bec36] [csclprd3-0-15:16499] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b8d3a6dddc0] [csclprd3-0-15:16499] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16499] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b8d3ab96cdd] [csclprd3-0-15:16499] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16499] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b16e8812270] [csclprd3-0-0:05919] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b16f0a1037e] [csclprd3-0-0:05919] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b16f18db515] [csclprd3-0-0:05919] [ 7] [csclprd3-0-15:16507] *** Process received signal *** [csclprd3-0-15:16507] Signal: Bus error (7) [csclprd3-0-15:16507] Signal code: Non-existant physical address (2) [csclprd3-0-15:16507] Failing at address: 0x2b14aa210f98 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b16e882cc36] [csclprd3-0-0:05919] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b16e884bdc0] [csclprd3-0-0:05919] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-0:05919] [10] [csclprd3-0-15:16511] *** Process received signal *** [csclprd3-0-15:16511] Signal: Bus error (7) [csclprd3-0-15:16511] Signal code: Non-existant physical address (2) [csclprd3-0-15:16511] Failing at address: 0x2b6eadd532b8 /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b16e8d04cdd] [csclprd3-0-0:05919] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-0:05919] *** End of error message *** [csclprd3-0-15:16513] *** Process received signal *** [csclprd3-0-15:16513] Signal: Bus error (7) [csclprd3-0-15:16513] Signal code: Non-existant physical address (2) [csclprd3-0-15:16513] Failing at address: 0x2b032ba6b3f8 [csclprd3-0-15:16513] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b031635a500] [csclprd3-0-15:16513] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b031972bf87] [csclprd3-0-15:16513] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b0319d35de8] [csclprd3-0-15:16513] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b031609406d] [csclprd3-0-15:16513] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b0316094270] [csclprd3-0-15:16513] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b031a13b37e] [csclprd3-0-15:16513] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b031b006515] [csclprd3-0-15:16513] [ 7] [csclprd3-0-15:16517] *** Process received signal *** [csclprd3-0-15:16517] Signal: Bus error (7) [csclprd3-0-15:16517] Signal code: Non-existant physical address (2) [csclprd3-0-15:16517] Failing at address: 0x2ac3eb9a5218 [csclprd3-0-15:16525] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab3fad8c500] [csclprd3-0-15:16525] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2ab3ff205e8a] [csclprd3-0-15:16525] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2ab3faac6169] [csclprd3-0-15:16525] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab3faac6270] [csclprd3-0-15:16525] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab3feb6d37e] [csclprd3-0-15:16525] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab3ffa38515] [csclprd3-0-15:16525] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab3faae0c36] [csclprd3-0-15:16525] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab3faaffdc0] [csclprd3-0-15:16525] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16525] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab3fafb8cdd] [csclprd3-0-15:16525] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16525] *** End of error message *** [csclprd3-0-15:16533] *** Process received signal *** [csclprd3-0-15:16533] Signal: Bus error (7) [csclprd3-0-15:16533] Signal code: Non-existant physical address (2) [csclprd3-0-15:16533] Failing at address: 0x2b83b700e358 [csclprd3-0-15:16527] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b2efdde4500] [csclprd3-0-15:16527] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b2f011b5f87] [csclprd3-0-15:16527] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b2f017bfde8] [csclprd3-0-15:16527] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b2efdb1e06d] [csclprd3-0-15:16527] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b2efdb1e270] [csclprd3-0-15:16527] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b2f01bc537e] [csclprd3-0-15:16527] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b2f02a90515] [csclprd3-0-15:16527] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b2efdb38c36] [csclprd3-0-15:16527] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b2efdb57dc0] [csclprd3-0-15:16527] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16527] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b2efe010cdd] [csclprd3-0-15:16527] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16527] *** End of error message *** [csclprd3-0-15:16545] *** Process received signal *** [csclprd3-0-15:16545] Signal: Bus error (7) [csclprd3-0-15:16545] Signal code: Non-existant physical address (2) [csclprd3-0-15:16545] Failing at address: 0x2b587d408498 [csclprd3-0-15:16545] [ 0] [csclprd3-0-15:16543] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab916715500] [csclprd3-0-15:16543] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab919ae6f87] [csclprd3-0-15:16543] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab91a0f0de8] [csclprd3-0-15:16543] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab91644f06d] [csclprd3-0-15:16543] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab91644f270] [csclprd3-0-15:16543] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab91a4f637e] [csclprd3-0-15:16543] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab91b3c1515] [csclprd3-0-15:16543] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab916469c36] [csclprd3-0-15:16543] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab916488dc0] [csclprd3-0-15:16543] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16543] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab916941cdd] [csclprd3-0-15:16543] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16543] *** End of error message *** [csclprd3-0-15:16482] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b07b6ff4500] [csclprd3-0-15:16482] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b07bb46de8a] [csclprd3-0-15:16482] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b07b6d2e169] [csclprd3-0-15:16482] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b07b6d2e270] [csclprd3-0-15:16482] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b07badd537e] [csclprd3-0-15:16482] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b07bbca0515] [csclprd3-0-15:16482] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b07b6d48c36] [csclprd3-0-15:16482] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b07b6d67dc0] [csclprd3-0-15:16482] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16482] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b07b7220cdd] [csclprd3-0-15:16482] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16482] *** End of error message *** [csclprd3-0-15:16487] *** Process received signal *** [csclprd3-0-15:16487] Signal: Bus error (7) [csclprd3-0-15:16487] Signal code: Non-existant physical address (2) [csclprd3-0-15:16487] Failing at address: 0x2b5ec1ccf178 [csclprd3-0-15:16487] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b5eac6a2500] [csclprd3-0-15:16487] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b5eafa73f87] [csclprd3-0-15:16487] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b5eb4204de8] [csclprd3-0-15:16487] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b5eac3dc06d] [csclprd3-0-15:16487] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b5eac3dc270] [csclprd3-0-15:16487] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b5eb460a37e] [csclprd3-0-15:16487] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b5eb54d5515] [csclprd3-0-15:16487] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b5eac3f6c36] [csclprd3-0-15:16487] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b5eac415dc0] [csclprd3-0-15:16487] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16487] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b5eac8cecdd] [csclprd3-0-15:16487] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16487] *** End of error message *** [csclprd3-0-15:16485] Signal: Bus error (7) [csclprd3-0-15:16485] Signal code: Non-existant physical address (2) [csclprd3-0-15:16485] Failing at address: 0x2b526f142180 [csclprd3-0-15:16485] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b5259bd5500] [csclprd3-0-15:16485] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b525e04ee8a] [csclprd3-0-15:16485] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b525990f169] [csclprd3-0-15:16485] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b525990f270] [csclprd3-0-15:16485] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b525d9b637e] [csclprd3-0-15:16485] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b525e881515] [csclprd3-0-15:16485] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b5259929c36] [csclprd3-0-15:16485] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b5259948dc0] [csclprd3-0-15:16485] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16485] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b5259e01cdd] [csclprd3-0-15:16485] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16485] *** End of error message *** [csclprd3-0-15:16489] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b9ef2ef5500] [csclprd3-0-15:16489] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b9ef736ee8a] [csclprd3-0-15:16489] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b9ef2c2f169] [csclprd3-0-15:16489] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b9ef2c2f270] [csclprd3-0-15:16489] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b9ef6cd637e] [csclprd3-0-15:16489] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b9ef7ba1515] [csclprd3-0-15:16489] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b9ef2c49c36] [csclprd3-0-15:16489] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b9ef2c68dc0] [csclprd3-0-15:16489] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16489] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b9ef3121cdd] [csclprd3-0-15:16489] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16489] *** End of error message *** [csclprd3-0-15:16539] *** Process received signal *** [csclprd3-0-15:16539] Signal: Bus error (7) [csclprd3-0-15:16539] Signal code: Non-existant physical address (2) [csclprd3-0-15:16539] Failing at address: 0x2ab07a7827b8 [csclprd3-0-15:16537] Signal: Bus error (7) [csclprd3-0-15:16537] Signal code: Non-existant physical address (2) [csclprd3-0-15:16537] Failing at address: 0x2acced2fe280 [csclprd3-0-15:16537] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2accd7ead500] [csclprd3-0-15:16537] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2acce0495e8a] [csclprd3-0-15:16537] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2accd7be7169] [csclprd3-0-15:16537] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2accd7be7270] [csclprd3-0-15:16537] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2accdbc8e37e] [csclprd3-0-15:16537] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2acce0cc8515] [csclprd3-0-15:16537] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2accd7c01c36] [csclprd3-0-15:16537] [ 7] /lib64/libpthread.so.0(+0xf500)[0x2b5867c7e500] [csclprd3-0-15:16545] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b586b04ff87] [csclprd3-0-15:16545] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b586b659de8] [csclprd3-0-15:16545] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b58679b806d] [csclprd3-0-15:16545] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b58679b8270] [csclprd3-0-15:16545] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b586ba5f37e] [csclprd3-0-15:16545] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b5870ac4515] [csclprd3-0-15:16545] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b58679d2c36] [csclprd3-0-15:16545] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b58679f1dc0] [csclprd3-0-15:16545] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16545] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b5867eaacdd] [csclprd3-0-15:16545] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16545] *** End of error message *** [csclprd3-0-15:16555] *** Process received signal *** [csclprd3-0-15:16555] Signal: Bus error (7) [csclprd3-0-15:16555] Signal code: Non-existant physical address (2) [csclprd3-0-15:16555] Failing at address: 0x2ade72d155d8 [csclprd3-0-15:16555] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ade5d373500] [csclprd3-0-15:16555] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ade60744f87] [csclprd3-0-15:16555] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ade60d4ede8] [csclprd3-0-15:16555] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ade5d0ad06d] [csclprd3-0-15:16555] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ade5d0ad270] [csclprd3-0-15:16555] [ 5] [csclprd3-0-15:16480] *** Process received signal *** [csclprd3-0-15:16480] Signal: Bus error (7) [csclprd3-0-15:16480] Signal code: Non-existant physical address (2) [csclprd3-0-15:16480] Failing at address: 0x2baa8907c718 [csclprd3-0-15:16480] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2baa738a4500] [csclprd3-0-15:16480] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2baa76c75f87] [csclprd3-0-15:16480] [csclprd3-0-15:16493] Signal: Bus error (7) [csclprd3-0-15:16493] Signal code: Non-existant physical address (2) [csclprd3-0-15:16493] Failing at address: 0x2ab17200b000 [csclprd3-0-15:16493] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab15cd01500] [csclprd3-0-15:16493] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2ab1652aae8a] [csclprd3-0-15:16493] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2ab15ca3b169] [csclprd3-0-15:16493] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab15ca3b270] [csclprd3-0-15:16493] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab164c1237e] [csclprd3-0-15:16493] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab165add515] [csclprd3-0-15:16493] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab15ca55c36] [csclprd3-0-15:16493] [ 7] [csclprd3-0-15:16503] *** Process received signal *** [csclprd3-0-15:16503] Signal: Bus error (7) [csclprd3-0-15:16503] Signal code: Non-existant physical address (2) [csclprd3-0-15:16503] Failing at address: 0x2ade494ce678 [csclprd3-0-15:16503] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ade33cbe500] [csclprd3-0-15:16503] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ade3708ff87] [csclprd3-0-15:16503] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ade37699de8] [csclprd3-0-15:16503] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ade339f806d] [csclprd3-0-15:16503] [ 4] [csclprd3-0-15:16507] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b1494d8e500] [csclprd3-0-15:16507] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b149c202f87] [csclprd3-0-15:16507] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b149c80cde8] [csclprd3-0-15:16507] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b1494ac806d] [csclprd3-0-15:16507] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b1494ac8270] [csclprd3-0-15:16507] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b149cc1237e] [csclprd3-0-15:16507] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b149dadd515] [csclprd3-0-15:16507] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b1494ae2c36] [csclprd3-0-15:16507] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b1494b01dc0] [csclprd3-0-15:16507] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16507] [10] [csclprd3-0-15:16511] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b6e9862a500] [csclprd3-0-15:16511] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b6e9b9fbf87] [csclprd3-0-15:16511] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b6ea0204de8] [csclprd3-0-15:16511] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b6e9836406d] [csclprd3-0-15:16511] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b6e98364270] [csclprd3-0-15:16511] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b6ea060a37e] [csclprd3-0-15:16511] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b6ea14d5515] [csclprd3-0-15:16511] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b6e9837ec36] [csclprd3-0-15:16511] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b6e9839ddc0] [csclprd3-0-15:16511] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16511] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6e98856cdd] [csclprd3-0-15:16511] [11] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b03160aec36] [csclprd3-0-15:16513] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b03160cddc0] [csclprd3-0-15:16513] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16513] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0316586cdd] [csclprd3-0-15:16513] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16513] *** End of error message *** [csclprd3-0-15:16519] *** Process received signal *** [csclprd3-0-15:16519] Signal: Bus error (7) [csclprd3-0-15:16519] Signal code: Non-existant physical address (2) [csclprd3-0-15:16519] Failing at address: 0x2ab8968d2538 [csclprd3-0-15:16519] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab88110c500] [csclprd3-0-15:16519] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab8844ddf87] [csclprd3-0-15:16519] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab884ae7de8] [csclprd3-0-15:16519] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab880e4606d] [csclprd3-0-15:16519] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab880e46270] [csclprd3-0-15:16519] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab884eed37e] [csclprd3-0-15:16519] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab885db8515] [csclprd3-0-15:16519] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab880e60c36] [csclprd3-0-15:16519] [ 8] [csclprd3-0-15:16517] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ac3d63ed500] [csclprd3-0-15:16517] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ac3d97bef87] [csclprd3-0-15:16517] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ac3d9dc8de8] [csclprd3-0-15:16517] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ac3d612706d] [csclprd3-0-15:16517] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ac3d6127270] [csclprd3-0-15:16517] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ac3da1ce37e] [csclprd3-0-15:16517] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ac3db099515] [csclprd3-0-15:16517] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ac3d6141c36] [csclprd3-0-15:16517] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ac3d6160dc0] [csclprd3-0-15:16517] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16517] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16511] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab880e7fdc0] [csclprd3-0-15:16519] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16519] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab881338cdd] [csclprd3-0-15:16519] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16519] *** End of error message *** [csclprd3-0-15:16530] *** Process received signal *** [csclprd3-0-15:16530] Signal: Bus error (7) [csclprd3-0-15:16530] Signal code: Non-existant physical address (2) [csclprd3-0-15:16530] Failing at address: 0x2ac2d78b5998 [csclprd3-0-15:16539] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab065047500] [csclprd3-0-15:16539] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab06c419f87] [csclprd3-0-15:16539] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab06ca23de8] [csclprd3-0-15:16539] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab064d8106d] [csclprd3-0-15:16539] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab064d81270] [csclprd3-0-15:16539] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab06ce2937e] [csclprd3-0-15:16539] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab06dcf4515] [csclprd3-0-15:16539] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2accd7c20dc0] [csclprd3-0-15:16537] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16537] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2accd80d9cdd] [csclprd3-0-15:16537] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16537] *** End of error message *** /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac3d6619cdd] [csclprd3-0-15:16517] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16517] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ade6115437e] [csclprd3-0-15:16555] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ade6201f515] [csclprd3-0-15:16555] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ade5d0c7c36] [csclprd3-0-15:16555] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ade5d0e6dc0] [csclprd3-0-15:16555] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16555] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ade5d59fcdd] [csclprd3-0-15:16555] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16555] *** End of error message *** [csclprd3-0-15:16481] Signal: Bus error (7) [csclprd3-0-15:16481] Signal code: Non-existant physical address (2) [csclprd3-0-15:16481] Failing at address: 0x2b250c997000 [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2baa7727fde8] [csclprd3-0-15:16480] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2baa735de06d] [csclprd3-0-15:16480] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2baa735de270] [csclprd3-0-15:16480] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2baa7768537e] [csclprd3-0-15:16480] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2baa7c630515] [csclprd3-0-15:16480] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2baa735f8c36] [csclprd3-0-15:16480] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2baa73617dc0] [csclprd3-0-15:16480] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16480] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2baa73ad0cdd] [csclprd3-0-15:16480] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16480] *** End of error message *** [csclprd3-0-15:16497] *** Process received signal *** [csclprd3-0-15:16497] Signal: Bus error (7) [csclprd3-0-15:16497] Signal code: Non-existant physical address (2) [csclprd3-0-15:16497] Failing at address: 0x2af4002ae8f8 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab15ca74dc0] [csclprd3-0-15:16493] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16493] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab15cf2dcdd] [csclprd3-0-15:16493] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16493] *** End of error message *** [csclprd3-0-15:16505] *** Process received signal *** [csclprd3-0-15:16505] Signal: Bus error (7) [csclprd3-0-15:16505] Signal code: Non-existant physical address (2) [csclprd3-0-15:16505] Failing at address: 0x2b845e1a9858 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ade339f8270] [csclprd3-0-15:16503] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ade37a9f37e] [csclprd3-0-15:16503] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ade3cac4515] [csclprd3-0-15:16503] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ade33a12c36] [csclprd3-0-15:16503] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ade33a31dc0] [csclprd3-0-15:16503] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16503] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ade33eeacdd] [csclprd3-0-15:16503] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16503] *** End of error message *** /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1494fbacdd] [csclprd3-0-15:16507] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16507] *** End of error message *** [csclprd3-0-15:16521] *** Process received signal *** [csclprd3-0-15:16521] Signal: Bus error (7) [csclprd3-0-15:16521] Signal code: Non-existant physical address (2) [csclprd3-0-15:16521] Failing at address: 0x2b426f6e5a38 [csclprd3-0-15:16530] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ac2c1ea8500] [csclprd3-0-15:16530] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ac2c5279f87] [csclprd3-0-15:16530] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ac2c5883de8] [csclprd3-0-15:16530] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ac2c1be206d] [csclprd3-0-15:16530] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ac2c1be2270] [csclprd3-0-15:16530] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ac2c5c8937e] [csclprd3-0-15:16530] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ac2c6b54515] [csclprd3-0-15:16530] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ac2c1bfcc36] [csclprd3-0-15:16530] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab064d9bc36] [csclprd3-0-15:16539] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab064dbadc0] [csclprd3-0-15:16539] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16539] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab065273cdd] [csclprd3-0-15:16539] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16539] *** End of error message *** [csclprd3-0-15:16481] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b24f757c500] [csclprd3-0-15:16481] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b24fb9f5e8a] [csclprd3-0-15:16481] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b24f72b6169] [csclprd3-0-15:16481] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b24f72b6270] [csclprd3-0-15:16481] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b24fb35d37e] [csclprd3-0-15:16481] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b2500428515] [csclprd3-0-15:16481] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b24f72d0c36] [csclprd3-0-15:16481] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b24f72efdc0] [csclprd3-0-15:16481] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16481] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b24f77a8cdd] [csclprd3-0-15:16481] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16481] *** End of error message *** [csclprd3-0-15:16497] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2af3ea961500] [csclprd3-0-15:16497] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2af3edd32f87] [csclprd3-0-15:16497] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2af3ee33cde8] [csclprd3-0-15:16497] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2af3ea69b06d] [csclprd3-0-15:16497] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2af3ea69b270] [csclprd3-0-15:16497] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2af3ee74237e] [csclprd3-0-15:16497] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2af3ef60d515] [csclprd3-0-15:16497] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2af3ea6b5c36] [csclprd3-0-15:16497] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2af3ea6d4dc0] [csclprd3-0-15:16497] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16497] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2af3eab8dcdd] [csclprd3-0-15:16497] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16497] *** End of error message *** [csclprd3-0-15:16505] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b84488dc500] [csclprd3-0-15:16505] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b844bcadf87] [csclprd3-0-15:16505] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b8450408de8] [csclprd3-0-15:16505] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b844861606d] [csclprd3-0-15:16505] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b8448616270] [csclprd3-0-15:16505] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b845080e37e] [csclprd3-0-15:16505] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b84516d9515] [csclprd3-0-15:16505] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b8448630c36] [csclprd3-0-15:16505] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b844864fdc0] [csclprd3-0-15:16505] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16505] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b8448b08cdd] [csclprd3-0-15:16505] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16505] *** End of error message *** [csclprd3-0-15:16523] *** Process received signal *** [csclprd3-0-15:16535] *** Process received signal *** [csclprd3-0-15:16535] Signal: Bus error (7) [csclprd3-0-15:16535] Signal code: Non-existant physical address (2) [csclprd3-0-15:16535] Failing at address: 0x2b7848154ad8 [csclprd3-0-15:16521] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b4259b38500] [csclprd3-0-15:16521] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b425cf09f87] [csclprd3-0-15:16521] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b425d513de8] [csclprd3-0-15:16521] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ac2c1c1bdc0] [csclprd3-0-15:16530] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16530] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac2c20d4cdd] [csclprd3-0-15:16530] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16530] *** End of error message *** [csclprd3-0-15:16491] *** Process received signal *** [csclprd3-0-15:16491] Signal: Bus error (7) [csclprd3-0-15:16491] Signal code: Non-existant physical address (2) [csclprd3-0-15:16491] Failing at address: 0x2b48298e2b78 [csclprd3-0-15:16523] Signal: Bus error (7) [csclprd3-0-15:16523] Signal code: Non-existant physical address (2) [csclprd3-0-15:16523] Failing at address: 0x2acee0a49c18 [csclprd3-0-15:16533] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b83a181e500] [csclprd3-0-15:16533] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b83a4beff87] [csclprd3-0-15:16533] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b83a51f9de8] [csclprd3-0-15:16533] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b83a155806d] [csclprd3-0-15:16533] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b83a1558270] [csclprd3-0-15:16533] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b83a55ff37e] [csclprd3-0-15:16533] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b83a64ca515] [csclprd3-0-15:16533] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b83a1572c36] [csclprd3-0-15:16533] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b83a1591dc0] [csclprd3-0-15:16533] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16533] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b83a1a4acdd] [csclprd3-0-15:16533] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16533] *** End of error message *** [csclprd3-0-15:16535] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b7832861500] [csclprd3-0-15:16535] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b7835c32f87] [csclprd3-0-15:16535] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b783623cde8] [csclprd3-0-15:16535] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b425987206d] [csclprd3-0-15:16521] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b4259872270] [csclprd3-0-15:16521] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b425d91937e] [csclprd3-0-15:16521] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b425e7e4515] [csclprd3-0-15:16521] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b425988cc36] [csclprd3-0-15:16521] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b42598abdc0] [csclprd3-0-15:16521] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16521] [10] [csclprd3-0-15:16491] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b4813e7e500] [csclprd3-0-15:16491] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b481724ff87] [csclprd3-0-15:16491] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b4817859de8] [csclprd3-0-15:16491] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b4813bb806d] [csclprd3-0-15:16491] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b4813bb8270] [csclprd3-0-15:16491] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b4817c5f37e] [csclprd3-0-15:16491] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b481ccc8515] [csclprd3-0-15:16491] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b4813bd2c36] [csclprd3-0-15:16491] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b4813bf1dc0] [csclprd3-0-15:16491] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16491] [10] [csclprd3-0-15:16549] Signal: Bus error (7) [csclprd3-0-15:16549] Signal code: Non-existant physical address (2) [csclprd3-0-15:16549] Failing at address: 0x2adf81bcf380 /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4259d64cdd] [csclprd3-0-15:16521] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16521] *** End of error message *** [csclprd3-0-15:16523] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2acecb09e500] [csclprd3-0-15:16523] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2acece46ff87] [csclprd3-0-15:16523] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2acecea79de8] [csclprd3-0-15:16523] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2acecadd806d] [csclprd3-0-15:16523] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2acecadd8270] [csclprd3-0-15:16523] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2acecee7f37e] [csclprd3-0-15:16523] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2acecfd4a515] [csclprd3-0-15:16523] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b783259b06d] [csclprd3-0-15:16535] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b783259b270] [csclprd3-0-15:16535] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b783664237e] [csclprd3-0-15:16535] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b783750d515] [csclprd3-0-15:16535] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b78325b5c36] [csclprd3-0-15:16535] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b78325d4dc0] [csclprd3-0-15:16535] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16535] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7832a8dcdd] [csclprd3-0-15:16535] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16535] *** End of error message *** /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b48140aacdd] [csclprd3-0-15:16491] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16491] *** End of error message *** [csclprd3-0-15:16509] *** Process received signal *** [csclprd3-0-15:16509] Signal: Bus error (7) [csclprd3-0-15:16509] Signal code: Non-existant physical address (2) [csclprd3-0-15:16509] Failing at address: 0x2b06b54d5cb8 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2acecadf2c36] [csclprd3-0-15:16523] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2acecae11dc0] [csclprd3-0-15:16523] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16523] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2acecb2cacdd] [csclprd3-0-15:16523] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16523] *** End of error message *** [csclprd3-0-15:16529] *** Process received signal *** [csclprd3-0-15:16529] Signal: Bus error (7) [csclprd3-0-15:16529] Signal code: Non-existant physical address (2) [csclprd3-0-15:16529] Failing at address: 0x2ac35742ed58 [csclprd3-0-15:16549] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2adf6c6bf500] [csclprd3-0-15:16549] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2adf74ca2e8a] [csclprd3-0-15:16549] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2adf6c3f9169] [csclprd3-0-15:16549] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2adf6c3f9270] [csclprd3-0-15:16549] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2adf7460a37e] [csclprd3-0-15:16549] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2adf754d5515] [csclprd3-0-15:16549] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2adf6c413c36] [csclprd3-0-15:16549] [ 7] [csclprd3-0-15:16509] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b069fa7b500] [csclprd3-0-15:16509] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b06a2e4cf87] [csclprd3-0-15:16509] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b06a3456de8] [csclprd3-0-15:16509] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2adf6c432dc0] [csclprd3-0-15:16549] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16549] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2adf6c8ebcdd] [csclprd3-0-15:16549] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16549] *** End of error message *** [csclprd3-0-15:16547] *** Process received signal *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b069f7b506d] [csclprd3-0-15:16509] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b069f7b5270] [csclprd3-0-15:16509] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b06a385c37e] [csclprd3-0-15:16509] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b06a8837515] [csclprd3-0-15:16509] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b069f7cfc36] [csclprd3-0-15:16509] [ 8] [csclprd3-0-15:16529] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ac3418cd500] [csclprd3-0-15:16529] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ac344c9ef87] [csclprd3-0-15:16529] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ac3452a8de8] [csclprd3-0-15:16529] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ac34160706d] [csclprd3-0-15:16529] [ 4] [csclprd3-0-15:16547] Signal: Bus error (7) [csclprd3-0-15:16547] Signal code: Non-existant physical address (2) [csclprd3-0-15:16547] Failing at address: 0x2ad3d2a0ddf8 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b069f7eedc0] [csclprd3-0-15:16509] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16509] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b069fca7cdd] [csclprd3-0-15:16509] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16509] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ac341607270] [csclprd3-0-15:16529] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ac3456ae37e] [csclprd3-0-15:16529] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ac346579515] [csclprd3-0-15:16529] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ac341621c36] [csclprd3-0-15:16529] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ac341640dc0] [csclprd3-0-15:16529] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16529] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac341af9cdd] [csclprd3-0-15:16529] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16529] *** End of error message *** [csclprd3-0-15:16547] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ad3bcf91500] [csclprd3-0-15:16547] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ad3c4410f87] [csclprd3-0-15:16547] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ad3c4a1ade8] [csclprd3-0-15:16547] [ 3] [csclprd3-0-15:16515] *** Process received signal *** [csclprd3-0-15:16515] Signal: Bus error (7) [csclprd3-0-15:16515] Signal code: Non-existant physical address (2) [csclprd3-0-15:16515] Failing at address: 0x2ab2e4fc7f38 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ad3bcccb06d] [csclprd3-0-15:16547] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ad3bcccb270] [csclprd3-0-15:16547] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ad3c4e2037e] [csclprd3-0-15:16547] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ad3c5ceb515] [csclprd3-0-15:16547] [ 7] [csclprd3-0-15:16479] *** Process received signal *** [csclprd3-0-15:16479] Signal: Bus error (7) [csclprd3-0-15:16479] Signal code: Non-existant physical address (2) [csclprd3-0-15:16479] Failing at address: 0x2ab745aa1080 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ad3bcce5c36] [csclprd3-0-15:16547] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ad3bcd04dc0] [csclprd3-0-15:16547] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16547] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad3bd1bdcdd] [csclprd3-0-15:16547] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16547] *** End of error message *** [csclprd3-0-15:16479] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab73a065500] [csclprd3-0-15:16479] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2ab73e4dee8a] [csclprd3-0-15:16479] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2ab739d9f169] [csclprd3-0-15:16479] [ 3] [csclprd3-0-15:16552] *** Process received signal *** [csclprd3-0-15:16552] Signal: Bus error (7) [csclprd3-0-15:16552] Signal code: Non-existant physical address (2) [csclprd3-0-15:16552] Failing at address: 0x2b0c966c3fd8 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab739d9f270] [csclprd3-0-15:16479] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab73de4637e] [csclprd3-0-15:16479] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab73ed11515] [csclprd3-0-15:16479] [ 6] [csclprd3-0-15:16515] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2ab2cf4c8500] [csclprd3-0-15:16515] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2ab2d2899f87] [csclprd3-0-15:16515] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2ab2d2ea3de8] [csclprd3-0-15:16515] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2ab2cf20206d] [csclprd3-0-15:16515] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2ab2cf202270] [csclprd3-0-15:16515] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2ab2d32a937e] [csclprd3-0-15:16515] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2ab2d8221515] [csclprd3-0-15:16515] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab739db9c36] [csclprd3-0-15:16479] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab739dd8dc0] [csclprd3-0-15:16479] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16479] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab73a291cdd] [csclprd3-0-15:16479] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16479] *** End of error message *** [csclprd3-0-15:16550] *** Process received signal *** [csclprd3-0-15:16550] Signal: Bus error (7) [csclprd3-0-15:16550] Signal code: Non-existant physical address (2) [csclprd3-0-15:16550] Failing at address: 0x2b6d42300078 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2ab2cf21cc36] [csclprd3-0-15:16515] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2ab2cf23bdc0] [csclprd3-0-15:16515] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16515] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab2cf6f4cdd] [csclprd3-0-15:16515] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16515] *** End of error message *** [csclprd3-0-15:16552] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b0c80ae5500] [csclprd3-0-15:16552] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b0c88000f87] [csclprd3-0-15:16552] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b0c8860ade8] [csclprd3-0-15:16552] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b0c8081f06d] [csclprd3-0-15:16552] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b0c8081f270] [csclprd3-0-15:16552] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b0c88a1037e] [csclprd3-0-15:16552] [ 6] [csclprd3-0-15:16541] *** Process received signal *** [csclprd3-0-15:16541] Signal: Bus error (7) [csclprd3-0-15:16541] Signal code: Non-existant physical address (2) [csclprd3-0-15:16541] Failing at address: 0x2b8c9d931118 /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b0c898db515] [csclprd3-0-15:16552] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b0c80839c36] [csclprd3-0-15:16552] [ 8] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b0c80858dc0] [csclprd3-0-15:16552] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16552] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0c80d11cdd] [csclprd3-0-15:16552] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16552] *** End of error message *** [csclprd3-0-15:16550] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b6d2c7f9500] [csclprd3-0-15:16550] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b6d2fbcaf87] [csclprd3-0-15:16550] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b6d34204de8] [csclprd3-0-15:16550] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b6d2c53306d] [csclprd3-0-15:16550] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b6d2c533270] [csclprd3-0-15:16550] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b6d3460a37e] [csclprd3-0-15:16550] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b6d354d5515] [csclprd3-0-15:16550] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b6d2c54dc36] [csclprd3-0-15:16550] [ 8] [csclprd3-0-15:16501] Signal: Bus error (7) [csclprd3-0-15:16501] Signal code: Non-existant physical address (2) [csclprd3-0-15:16501] Failing at address: 0x2b27f9135080 /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b6d2c56cdc0] [csclprd3-0-15:16550] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16550] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6d2ca25cdd] [csclprd3-0-15:16550] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16550] *** End of error message *** [csclprd3-0-15:16541] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b8c87d9c500] [csclprd3-0-15:16541] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_allocator_bucket.so(mca_allocator_bucket_alloc_align+0x97)[0x2b8c8b16df87] [csclprd3-0-15:16541] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_mpool_sm.so(mca_mpool_sm_alloc+0x18)[0x2b8c8b777de8] [csclprd3-0-15:16541] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x11d)[0x2b8c87ad606d] [csclprd3-0-15:16541] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b8c87ad6270] [csclprd3-0-15:16541] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b8c8bb7d37e] [csclprd3-0-15:16541] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b8c90ac4515] [csclprd3-0-15:16541] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b8c87af0c36] [csclprd3-0-15:16541] [ 8] [csclprd3-0-15:16501] [ 0] /lib64/libpthread.so.0(+0xf500)[0x2b27e3ea1500] [csclprd3-0-15:16501] [ 1] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_btl_sm.so(+0x4e8a)[0x2b27ec495e8a] [csclprd3-0-15:16501] [ 2] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_grow+0x219)[0x2b27e3bdb169] [csclprd3-0-15:16501] [ 3] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x2b27e3bdb270] [csclprd3-0-15:16501] [ 4] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_bml_r2.so(+0x137e)[0x2b27e7c8237e] [csclprd3-0-15:16501] [ 5] /hpc/apps/mpi/openmpi/1.8.7/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x2b27eccc8515] [csclprd3-0-15:16501] [ 6] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b8c87b0fdc0] [csclprd3-0-15:16541] [ 9] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16541] [10] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b8c87fc8cdd] [csclprd3-0-15:16541] [11] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16541] *** End of error message *** /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x2b27e3bf5c36] [csclprd3-0-15:16501] [ 7] /hpc/apps/mpi/openmpi/1.8.7/lib/libmpi.so.1(MPI_Init+0x170)[0x2b27e3c14dc0] [csclprd3-0-15:16501] [ 8] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400ad0] [csclprd3-0-15:16501] [ 9] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b27e40cdcdd] [csclprd3-0-15:16501] [10] /hpc/home/lanew/mpi/openmpi/ProcessColors3[0x400999] [csclprd3-0-15:16501] *** End of error message *** -------------------------------------------------------------------------- WARNING: A process refused to die despite all the efforts! This process may still be running and/or consuming resources. Host: csclprd3-4-3 PID: 13849 -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun noticed that process rank 53 with PID 0 on node csclprd3-0-0 exited on signal 7 (Bus error). --------------------------------------------------------------------------
