This will be in the next nightly 1.8.5 tarball. Bill: can you test it to see if we’ve fixed the problem?
Thanks Ralph > On Apr 10, 2015, at 2:15 PM, Ralph Castain <r...@open-mpi.org> wrote: > > Okay, I at least now understand the behavior from this particular cmd line. > Looks like we are binding-to-core by default, even if you specify > use-hwthread-cpus. I’ll fix that one - still don’t understand the segfaults. > > Bill: can you shed some light on those? > > >> On Apr 9, 2015, at 8:28 PM, Lane, William <william.l...@cshs.org >> <mailto:william.l...@cshs.org>> wrote: >> >> Ralph, >> >> In looking at the /proc/cpuinfo textfile it looks like hyperthreading >> is enabled (in that it indicates 16 siblings for each of the 8 cores of the >> two LGA2011 CPU's). I don't have access to the BIOS on this system though >> so I'll have to check w/someone else. >> >> I have done more testing and found that at 104 slots requested OpenMPI >> won't run the LAPACK benchmark. All the LGA2011 nodes exhibit the same >> strange binding behavior (maybe because hyperthreading is turned on for >> these nodes, but no the LGA 1366 nodes?). Below is all the relevant >> information to >> that run: >> >> II. >> a. $MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile >> hostfile-single --mca btl_tcp_if_include eth0 --hetero-nodes >> --use-hwthread-cpus --prefix $MPI_DIR $BENCH_DIR/$APP_DIR/$APP_BIN >> >> where NSLOTS=104 >> >> b. >> [lanew@csclprd3s1 hpl]$ . /hpc/apps/benchmarks/runhpl4.job >> [csclprd3-6-1:27586] MCW rank 3 bound to socket 1[core 3[hwt 0]]: [./.][./B] >> [csclprd3-6-1:27586] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/.][./.] >> [csclprd3-6-1:27586] MCW rank 1 bound to socket 1[core 2[hwt 0]]: [./.][B/.] >> [csclprd3-6-1:27586] MCW rank 2 bound to socket 0[core 1[hwt 0]]: [./B][./.] >> [csclprd3-0-2:04454] MCW rank 27 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-2:04454] MCW rank 28 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-2:04454] MCW rank 29 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-2:04454] MCW rank 30 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-2:04454] MCW rank 31 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-2:04454] MCW rank 26 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-0:21129] MCW rank 8 bound to socket 0[core 0[hwt 0]]: >> [B/././././.][./././././.] >> [csclprd3-0-0:21129] MCW rank 9 bound to socket 1[core 6[hwt 0]]: >> [./././././.][B/././././.] >> [csclprd3-0-0:21129] MCW rank 10 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.][./././././.] >> [csclprd3-0-0:21129] MCW rank 11 bound to socket 1[core 7[hwt 0]]: >> [./././././.][./B/./././.] >> [csclprd3-0-0:21129] MCW rank 12 bound to socket 0[core 2[hwt 0]]: >> [././B/././.][./././././.] >> [csclprd3-0-0:21129] MCW rank 13 bound to socket 1[core 8[hwt 0]]: >> [./././././.][././B/././.] >> [csclprd3-0-0:21129] MCW rank 14 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.][./././././.] >> [csclprd3-0-0:21129] MCW rank 15 bound to socket 1[core 9[hwt 0]]: >> [./././././.][./././B/./.] >> [csclprd3-0-0:21129] MCW rank 16 bound to socket 0[core 4[hwt 0]]: >> [././././B/.][./././././.] >> [csclprd3-0-0:21129] MCW rank 17 bound to socket 1[core 10[hwt 0]]: >> [./././././.][././././B/.] >> [csclprd3-0-0:21129] MCW rank 18 bound to socket 0[core 5[hwt 0]]: >> [./././././B][./././././.] >> [csclprd3-0-0:21129] MCW rank 19 bound to socket 1[core 11[hwt 0]]: >> [./././././.][./././././B] >> [csclprd3-0-1:12882] MCW rank 22 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-1:12882] MCW rank 23 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-1:12882] MCW rank 24 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-1:12882] MCW rank 25 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-1:12882] MCW rank 20 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-1:12882] MCW rank 21 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-9:27268] MCW rank 92 bound to socket 0[core 2[hwt 0-1]]: >> [../../BB/../../../../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 93 bound to socket 1[core 10[hwt 0-1]]: >> [../../../../../../../..][../../BB/../../../../..] >> [csclprd3-0-9:27268] MCW rank 94 bound to socket 0[core 3[hwt 0-1]]: >> [../../../BB/../../../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 95 bound to socket 1[core 11[hwt 0-1]]: >> [../../../../../../../..][../../../BB/../../../..] >> [csclprd3-0-9:27268] MCW rank 96 bound to socket 0[core 4[hwt 0-1]]: >> [../../../../BB/../../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 97 bound to socket 1[core 12[hwt 0-1]]: >> [../../../../../../../..][../../../../BB/../../..] >> [csclprd3-0-8:22880] MCW rank 84 bound to socket 0[core 6[hwt 0-1]]: >> [../../../../../../BB/..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 85 bound to socket 1[core 14[hwt 0-1]]: >> [../../../../../../../..][../../../../../../BB/..] >> [csclprd3-0-8:22880] MCW rank 86 bound to socket 0[core 7[hwt 0-1]]: >> [../../../../../../../BB][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 87 bound to socket 1[core 15[hwt 0-1]]: >> [../../../../../../../..][../../../../../../../BB] >> [csclprd3-0-8:22880] MCW rank 72 bound to socket 0[core 0[hwt 0-1]]: >> [BB/../../../../../../..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 73 bound to socket 1[core 8[hwt 0-1]]: >> [../../../../../../../..][BB/../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 74 bound to socket 0[core 1[hwt 0-1]]: >> [../BB/../../../../../..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 75 bound to socket 1[core 9[hwt 0-1]]: >> [../../../../../../../..][../BB/../../../../../..] >> [csclprd3-0-8:22880] MCW rank 76 bound to socket 0[core 2[hwt 0-1]]: >> [../../BB/../../../../..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 77 bound to socket 1[core 10[hwt 0-1]]: >> [../../../../../../../..][../../BB/../../../../..] >> [csclprd3-0-8:22880] MCW rank 78 bound to socket 0[core 3[hwt 0-1]]: >> [../../../BB/../../../..][../../../../../../../..] >> [csclprd3-6-5:18000] MCW rank 7 bound to socket 1[core 3[hwt 0]]: [./.][./B] >> [csclprd3-6-5:18000] MCW rank 4 bound to socket 0[core 0[hwt 0]]: [B/.][./.] >> [csclprd3-6-5:18000] MCW rank 5 bound to socket 1[core 2[hwt 0]]: [./.][B/.] >> [csclprd3-6-5:18000] MCW rank 6 bound to socket 0[core 1[hwt 0]]: [./B][./.] >> [csclprd3-0-7:08058] MCW rank 60 bound to socket 0[core 2[hwt 0-1]]: >> [../../BB/../../../../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 61 bound to socket 1[core 10[hwt 0-1]]: >> [../../../../../../../..][../../BB/../../../../..] >> [csclprd3-0-7:08058] MCW rank 62 bound to socket 0[core 3[hwt 0-1]]: >> [../../../BB/../../../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 63 bound to socket 1[core 11[hwt 0-1]]: >> [../../../../../../../..][../../../BB/../../../..] >> [csclprd3-0-7:08058] MCW rank 64 bound to socket 0[core 4[hwt 0-1]]: >> [../../../../BB/../../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 65 bound to socket 1[core 12[hwt 0-1]]: >> [../../../../../../../..][../../../../BB/../../..] >> [csclprd3-0-7:08058] MCW rank 66 bound to socket 0[core 5[hwt 0-1]]: >> [../../../../../BB/../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 67 bound to socket 1[core 13[hwt 0-1]]: >> [../../../../../../../..][../../../../../BB/../..] >> [csclprd3-0-7:08058] MCW rank 68 bound to socket 0[core 6[hwt 0-1]]: >> [../../../../../../BB/..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 69 bound to socket 1[core 14[hwt 0-1]]: >> [../../../../../../../..][../../../../../../BB/..] >> [csclprd3-0-7:08058] MCW rank 70 bound to socket 0[core 7[hwt 0-1]]: >> [../../../../../../../BB][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 71 bound to socket 1[core 15[hwt 0-1]]: >> [../../../../../../../..][../../../../../../../BB] >> [csclprd3-0-7:08058] MCW rank 56 bound to socket 0[core 0[hwt 0-1]]: >> [BB/../../../../../../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 57 bound to socket 1[core 8[hwt 0-1]]: >> [../../../../../../../..][BB/../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 58 bound to socket 0[core 1[hwt 0-1]]: >> [../BB/../../../../../..][../../../../../../../..] >> [csclprd3-0-7:08058] MCW rank 59 bound to socket 1[core 9[hwt 0-1]]: >> [../../../../../../../..][../BB/../../../../../..] >> [csclprd3-0-5:15446] MCW rank 44 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-5:15446] MCW rank 45 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-5:15446] MCW rank 46 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-5:15446] MCW rank 47 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-5:15446] MCW rank 48 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-5:15446] MCW rank 49 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-9:27268] MCW rank 98 bound to socket 0[core 5[hwt 0-1]]: >> [../../../../../BB/../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 99 bound to socket 1[core 13[hwt 0-1]]: >> [../../../../../../../..][../../../../../BB/../..] >> [csclprd3-0-9:27268] MCW rank 100 bound to socket 0[core 6[hwt 0-1]]: >> [../../../../../../BB/..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 101 bound to socket 1[core 14[hwt 0-1]]: >> [../../../../../../../..][../../../../../../BB/..] >> [csclprd3-0-9:27268] MCW rank 102 bound to socket 0[core 7[hwt 0-1]]: >> [../../../../../../../BB][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 103 bound to socket 1[core 15[hwt 0-1]]: >> [../../../../../../../..][../../../../../../../BB] >> [csclprd3-0-9:27268] MCW rank 88 bound to socket 0[core 0[hwt 0-1]]: >> [BB/../../../../../../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 89 bound to socket 1[core 8[hwt 0-1]]: >> [../../../../../../../..][BB/../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 90 bound to socket 0[core 1[hwt 0-1]]: >> [../BB/../../../../../..][../../../../../../../..] >> [csclprd3-0-9:27268] MCW rank 91 bound to socket 1[core 9[hwt 0-1]]: >> [../../../../../../../..][../BB/../../../../../..] >> [csclprd3-0-6:28854] MCW rank 51 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-6:28854] MCW rank 52 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-6:28854] MCW rank 53 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-6:28854] MCW rank 54 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-6:28854] MCW rank 55 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-6:28854] MCW rank 50 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-4:28072] MCW rank 38 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-4:28072] MCW rank 39 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-4:28072] MCW rank 40 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-4:28072] MCW rank 41 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-4:28072] MCW rank 42 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-4:28072] MCW rank 43 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-8:22880] MCW rank 79 bound to socket 1[core 11[hwt 0-1]]: >> [../../../../../../../..][../../../BB/../../../..] >> [csclprd3-0-8:22880] MCW rank 80 bound to socket 0[core 4[hwt 0-1]]: >> [../../../../BB/../../..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 81 bound to socket 1[core 12[hwt 0-1]]: >> [../../../../../../../..][../../../../BB/../../..] >> [csclprd3-0-8:22880] MCW rank 82 bound to socket 0[core 5[hwt 0-1]]: >> [../../../../../BB/../..][../../../../../../../..] >> [csclprd3-0-8:22880] MCW rank 83 bound to socket 1[core 13[hwt 0-1]]: >> [../../../../../../../..][../../../../../BB/../..] >> [csclprd3-0-3:11837] MCW rank 33 bound to socket 0[core 1[hwt 0]]: >> [./B/./././.] >> [csclprd3-0-3:11837] MCW rank 34 bound to socket 0[core 2[hwt 0]]: >> [././B/././.] >> [csclprd3-0-3:11837] MCW rank 35 bound to socket 0[core 3[hwt 0]]: >> [./././B/./.] >> [csclprd3-0-3:11837] MCW rank 36 bound to socket 0[core 4[hwt 0]]: >> [././././B/.] >> [csclprd3-0-3:11837] MCW rank 37 bound to socket 0[core 5[hwt 0]]: >> [./././././B] >> [csclprd3-0-3:11837] MCW rank 32 bound to socket 0[core 0[hwt 0]]: >> [B/././././.] >> [csclprd3-0-9:27275] *** Process received signal *** >> [csclprd3-0-9:27275] Signal: Bus error (7) >> [csclprd3-0-9:27275] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27275] Failing at address: 0x7f1215253000 >> [csclprd3-0-9:27276] *** Process received signal *** >> [csclprd3-0-9:27276] Signal: Bus error (7) >> [csclprd3-0-9:27276] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27276] Failing at address: 0x7f563e874380 >> [csclprd3-0-9:27277] *** Process received signal *** >> [csclprd3-0-9:27277] Signal: Bus error (7) >> [csclprd3-0-9:27277] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27277] Failing at address: 0x7fbbec79a300 >> [csclprd3-0-9:27278] *** Process received signal *** >> [csclprd3-0-9:27278] Signal: Bus error (7) >> [csclprd3-0-9:27278] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27278] Failing at address: 0x7fbadf816080 >> [csclprd3-0-9:27279] *** Process received signal *** >> [csclprd3-0-9:27279] Signal: Bus error (7) >> [csclprd3-0-9:27279] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27279] Failing at address: 0x7fab5dfa0100 >> [csclprd3-0-9:27280] *** Process received signal *** >> [csclprd3-0-9:27280] Signal: Bus error (7) >> [csclprd3-0-9:27280] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27280] Failing at address: 0x7f0bb4034500 >> [csclprd3-0-9:27281] *** Process received signal *** >> [csclprd3-0-9:27281] Signal: Bus error (7) >> [csclprd3-0-9:27281] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27281] Failing at address: 0x7f49bb544f80 >> [csclprd3-0-9:27282] *** Process received signal *** >> [csclprd3-0-9:27282] Signal: Bus error (7) >> [csclprd3-0-9:27282] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27282] Failing at address: 0x7fe647f61f00 >> [csclprd3-0-9:27283] *** Process received signal *** >> [csclprd3-0-9:27283] Signal: Bus error (7) >> [csclprd3-0-9:27283] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27283] Failing at address: 0x7f79a9d25580 >> [csclprd3-0-9:27272] *** Process received signal *** >> [csclprd3-0-9:27272] Signal: Bus error (7) >> [csclprd3-0-9:27272] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27272] Failing at address: 0x7f64568adf80 >> [csclprd3-0-9:27269] *** Process received signal *** >> [csclprd3-0-9:27269] Signal: Bus error (7) >> [csclprd3-0-9:27269] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27269] Failing at address: 0x7f5e2a17e580 >> [csclprd3-0-9:27270] *** Process received signal *** >> [csclprd3-0-9:27270] Signal: Bus error (7) >> [csclprd3-0-9:27270] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27270] Failing at address: 0x7fda95421400 >> [csclprd3-0-9:27271] *** Process received signal *** >> [csclprd3-0-9:27271] Signal: Bus error (7) >> [csclprd3-0-9:27271] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27271] Failing at address: 0x7f873e76c100 >> [csclprd3-0-9:27271] [ 0] [csclprd3-0-9:27273] *** Process received signal >> *** >> [csclprd3-0-9:27273] Signal: Bus error (7) >> [csclprd3-0-9:27273] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27273] Failing at address: 0x7f5dc6e99e80 >> [csclprd3-0-9:27274] *** Process received signal *** >> [csclprd3-0-9:27274] Signal: Bus error (7) >> [csclprd3-0-9:27274] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27274] Failing at address: 0x7f83afce2280 >> [csclprd3-0-9:27274] [ 0] [csclprd3-0-9:27269] [ 0] >> /lib64/libc.so.6(+0x32920)[0x7f5e39b82920] >> [csclprd3-0-9:27269] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f5e2f5f111a] >> [csclprd3-0-9:27269] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f5e3a6960c9] >> [csclprd3-0-9:27269] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f5e3a696200] >> [csclprd3-0-9:27269] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f5e2f9f838e] >> [csclprd3-0-9:27269] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f5e2eb204e5] >> [csclprd3-0-9:27269] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f5e3a6b0e26] >> [csclprd3-0-9:27269] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f5e3a6cfe10] >> [csclprd3-0-9:27269] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27269] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5e39b6ecdd] >> [csclprd3-0-9:27269] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27269] *** End of error message *** >> [csclprd3-0-9:27270] [ 0] /lib64/libc.so.6(+0x32920)[0x7fdaa8d89920] >> [csclprd3-0-9:27270] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7fdaa29d711a] >> [csclprd3-0-9:27270] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7fdaa989d0c9] >> [csclprd3-0-9:27270] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7fdaa989d200] >> [csclprd3-0-9:27270] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7fdaa2dde38e] >> [csclprd3-0-9:27270] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7fdaa1f064e5] >> [csclprd3-0-9:27270] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7fdaa98b7e26] >> [csclprd3-0-9:27270] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7fdaa98d6e10] >> [csclprd3-0-9:27270] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27270] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fdaa8d75cdd] >> [csclprd3-0-9:27270] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27270] *** End of error message *** >> /lib64/libc.so.6(+0x32920)[0x7f875211a920] >> [csclprd3-0-9:27271] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f8747dfe11a] >> [csclprd3-0-9:27271] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f8752c2e0c9] >> [csclprd3-0-9:27271] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f8752c2e200] >> [csclprd3-0-9:27271] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f874c20538e] >> [csclprd3-0-9:27271] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f874732d4e5] >> [csclprd3-0-9:27271] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f8752c48e26] >> [csclprd3-0-9:27271] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f8752c67e10] >> [csclprd3-0-9:27271] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27271] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f8752106cdd] >> [csclprd3-0-9:27271] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27271] *** End of error message *** >> [csclprd3-0-9:27273] [ 0] /lib64/libc.so.6(+0x32920)[0x7f5ddaa3c920] >> [csclprd3-0-9:27273] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f5dd47ae11a] >> [csclprd3-0-9:27273] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f5ddb5500c9] >> [csclprd3-0-9:27273] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f5ddb550200] >> [csclprd3-0-9:27273] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f5dd4bb538e] >> [csclprd3-0-9:27273] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f5dcfbe54e5] >> [csclprd3-0-9:27273] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f5ddb56ae26] >> [csclprd3-0-9:27273] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f5ddb589e10] >> [csclprd3-0-9:27273] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27273] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5ddaa28cdd] >> [csclprd3-0-9:27273] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27273] *** End of error message *** >> [csclprd3-0-9:27275] [ 0] /lib64/libc.so.6(+0x32920)[0x7f1228ede920] >> [csclprd3-0-9:27275] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f1222bac11a] >> [csclprd3-0-9:27275] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f12299f20c9] >> [csclprd3-0-9:27275] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f12299f2200] >> [csclprd3-0-9:27275] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f1222fb338e] >> [csclprd3-0-9:27275] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f12220db4e5] >> [csclprd3-0-9:27275] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f1229a0ce26] >> [csclprd3-0-9:27275] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f1229a2be10] >> [csclprd3-0-9:27275] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27275] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f1228ecacdd] >> [csclprd3-0-9:27275] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27275] *** End of error message *** >> [csclprd3-0-9:27276] [ 0] /lib64/libc.so.6(+0x32920)[0x7f565218a920] >> [csclprd3-0-9:27276] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f5647dfe11a] >> [csclprd3-0-9:27276] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f5652c9e0c9] >> [csclprd3-0-9:27276] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f5652c9e200] >> [csclprd3-0-9:27276] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f564c2f738e] >> [csclprd3-0-9:27276] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f564732d4e5] >> [csclprd3-0-9:27276] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f5652cb8e26] >> [csclprd3-0-9:27276] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f5652cd7e10] >> [csclprd3-0-9:27276] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27276] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f5652176cdd] >> [csclprd3-0-9:27276] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27276] *** End of error message *** >> [csclprd3-0-9:27277] [ 0] /lib64/libc.so.6(+0x32920)[0x7fbc00059920] >> [csclprd3-0-9:27277] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7fbbf9de811a] >> [csclprd3-0-9:27277] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7fbc00b6d0c9] >> [csclprd3-0-9:27277] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7fbc00b6d200] >> [csclprd3-0-9:27277] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7fbbfa1ef38e] >> [csclprd3-0-9:27277] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7fbbf93174e5] >> [csclprd3-0-9:27277] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7fbc00b87e26] >> [csclprd3-0-9:27277] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7fbc00ba6e10] >> [csclprd3-0-9:27277] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27277] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fbc00045cdd] >> [csclprd3-0-9:27277] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27277] *** End of error message *** >> [csclprd3-0-9:27278] [ 0] /lib64/libc.so.6(+0x32920)[0x7fbaf33a1920] >> [csclprd3-0-9:27278] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7fbaed0a211a] >> [csclprd3-0-9:27278] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7fbaf3eb50c9] >> [csclprd3-0-9:27278] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7fbaf3eb5200] >> [csclprd3-0-9:27278] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7fbaed4a938e] >> [csclprd3-0-9:27278] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7fbaec5d14e5] >> [csclprd3-0-9:27278] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7fbaf3ecfe26] >> [csclprd3-0-9:27278] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7fbaf3eeee10] >> [csclprd3-0-9:27278] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27278] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fbaf338dcdd] >> [csclprd3-0-9:27278] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27278] *** End of error message *** >> [csclprd3-0-9:27279] [ 0] /lib64/libc.so.6(+0x32920)[0x7fab71930920] >> [csclprd3-0-9:27279] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7fab675f111a] >> [csclprd3-0-9:27279] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7fab724440c9] >> [csclprd3-0-9:27279] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7fab72444200] >> [csclprd3-0-9:27279] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7fab679f838e] >> [csclprd3-0-9:27279] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7fab66b204e5] >> [csclprd3-0-9:27279] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7fab7245ee26] >> [csclprd3-0-9:27279] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7fab7247de10] >> [csclprd3-0-9:27279] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27279] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fab7191ccdd] >> [csclprd3-0-9:27279] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27279] *** End of error message *** >> [csclprd3-0-9:27280] [ 0] /lib64/libc.so.6(+0x32920)[0x7f0bc7a18920] >> [csclprd3-0-9:27280] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f0bc163b11a] >> [csclprd3-0-9:27280] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f0bc852c0c9] >> [csclprd3-0-9:27280] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f0bc852c200] >> [csclprd3-0-9:27280] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f0bc1a4238e] >> [csclprd3-0-9:27280] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f0bc0b6a4e5] >> [csclprd3-0-9:27280] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f0bc8546e26] >> [csclprd3-0-9:27280] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f0bc8565e10] >> [csclprd3-0-9:27280] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27280] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f0bc7a04cdd] >> [csclprd3-0-9:27280] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27280] *** End of error message *** >> [csclprd3-0-9:27281] [ 0] /lib64/libc.so.6(+0x32920)[0x7f49cf009920] >> [csclprd3-0-9:27281] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f49c8d0911a] >> [csclprd3-0-9:27281] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f49cfb1d0c9] >> [csclprd3-0-9:27281] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f49cfb1d200] >> [csclprd3-0-9:27281] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f49c911038e] >> [csclprd3-0-9:27281] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f49c82384e5] >> [csclprd3-0-9:27281] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f49cfb37e26] >> [csclprd3-0-9:27281] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f49cfb56e10] >> [csclprd3-0-9:27281] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27281] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f49ceff5cdd] >> [csclprd3-0-9:27281] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27281] *** End of error message *** >> [csclprd3-0-9:27282] [ 0] /lib64/libc.so.6(+0x32920)[0x7fe65bb89920] >> [csclprd3-0-9:27282] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7fe6557b711a] >> [csclprd3-0-9:27282] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7fe65c69d0c9] >> [csclprd3-0-9:27282] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7fe65c69d200] >> [csclprd3-0-9:27282] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7fe655bbe38e] >> [csclprd3-0-9:27282] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7fe654ce64e5] >> [csclprd3-0-9:27282] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7fe65c6b7e26] >> [csclprd3-0-9:27282] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7fe65c6d6e10] >> [csclprd3-0-9:27282] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27282] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7fe65bb75cdd] >> [csclprd3-0-9:27282] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27282] *** End of error message *** >> [csclprd3-0-9:27283] [ 0] /lib64/libc.so.6(+0x32920)[0x7f79bd430920] >> [csclprd3-0-9:27283] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f79b31e911a] >> [csclprd3-0-9:27283] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f79bdf440c9] >> [csclprd3-0-9:27283] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f79bdf44200] >> [csclprd3-0-9:27283] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f79b35f038e] >> [csclprd3-0-9:27283] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f79b27184e5] >> [csclprd3-0-9:27283] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f79bdf5ee26] >> [csclprd3-0-9:27283] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f79bdf7de10] >> [csclprd3-0-9:27283] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27283] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f79bd41ccdd] >> [csclprd3-0-9:27283] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27283] *** End of error message *** >> /lib64/libc.so.6(+0x32920)[0x7f83c367f920] >> [csclprd3-0-9:27274] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f83bd2f211a] >> [csclprd3-0-9:27274] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f83c41930c9] >> [csclprd3-0-9:27274] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f83c4193200] >> [csclprd3-0-9:27274] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f83bd6f938e] >> [csclprd3-0-9:27274] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f83bc8214e5] >> [csclprd3-0-9:27274] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f83c41ade26] >> [csclprd3-0-9:27274] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f83c41cce10] >> [csclprd3-0-9:27274] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27274] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f83c366bcdd] >> [csclprd3-0-9:27274] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27274] *** End of error message *** >> [csclprd3-0-9:27284] *** Process received signal *** >> [csclprd3-0-9:27284] Signal: Bus error (7) >> [csclprd3-0-9:27284] Signal code: Non-existant physical address (2) >> [csclprd3-0-9:27284] Failing at address: 0x7f7273480e80 >> [csclprd3-0-9:27284] [ 0] /lib64/libc.so.6(+0x32920)[0x7f7287160920] >> [csclprd3-0-9:27284] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f7280ebf11a] >> [csclprd3-0-9:27284] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f7287c740c9] >> [csclprd3-0-9:27284] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f7287c74200] >> [csclprd3-0-9:27284] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f72812c638e] >> [csclprd3-0-9:27284] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f72803ee4e5] >> [csclprd3-0-9:27284] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f7287c8ee26] >> [csclprd3-0-9:27284] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f7287cade10] >> [csclprd3-0-9:27284] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27284] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f728714ccdd] >> [csclprd3-0-9:27284] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27284] *** End of error message *** >> [csclprd3-0-9:27272] [ 0] /lib64/libc.so.6(+0x32920)[0x7f646a415920] >> [csclprd3-0-9:27272] [ 1] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_btl_sm.so(+0x511a)[0x7f64641b911a] >> [csclprd3-0-9:27272] [ 2] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_grow+0x239)[0x7f646af290c9] >> [csclprd3-0-9:27272] [ 3] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_free_list_resize_mt+0x40)[0x7f646af29200] >> [csclprd3-0-9:27272] [ 4] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_bml_r2.so(+0x138e)[0x7f64645c038e] >> [csclprd3-0-9:27272] [ 5] >> /hpc/apps/mpi/openmpi/1.8.2/lib/openmpi/mca_pml_ob1.so(mca_pml_ob1_add_procs+0xd5)[0x7f645f5344e5] >> [csclprd3-0-9:27272] [ 6] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(ompi_mpi_init+0x8d6)[0x7f646af43e26] >> [csclprd3-0-9:27272] [ 7] >> /hpc/apps/mpi/openmpi/1.8.2/lib/libmpi.so.1(MPI_Init+0x170)[0x7f646af62e10] >> [csclprd3-0-9:27272] [ 8] /hpc/apps/benchmarks/hpl/xhpl[0x401571] >> [csclprd3-0-9:27272] [ 9] >> /lib64/libc.so.6(__libc_start_main+0xfd)[0x7f646a401cdd] >> [csclprd3-0-9:27272] [10] /hpc/apps/benchmarks/hpl/xhpl[0x401439] >> [csclprd3-0-9:27272] *** End of error message *** >> -------------------------------------------------------------------------- >> mpirun noticed that process rank 88 with PID 27269 on node csclprd3-0-9 >> exited on signal 7 (Bus error). >> -------------------------------------------------------------------------- >> 16 total processes killed (some possibly by mpirun during cleanup) >> >> c. hostfile >> csclprd3-6-1 slots=4 max-slots=4 >> csclprd3-6-5 slots=4 max-slots=4 >> csclprd3-0-0 slots=12 max-slots=24 >> csclprd3-0-1 slots=6 max-slots=12 >> csclprd3-0-2 slots=6 max-slots=12 >> csclprd3-0-3 slots=6 max-slots=12 >> csclprd3-0-4 slots=6 max-slots=12 >> csclprd3-0-5 slots=6 max-slots=12 >> csclprd3-0-6 slots=6 max-slots=12 >> #total number of successfully tested non-hyperthreaded computes slots at >> this point is 56 >> csclprd3-0-7 slots=16 max-slots=32 >> #total number of successfully tested slots at this point is 72 >> csclprd3-0-8 slots=16 max-slots=32 >> #total number of successfully tested slots at this point is 88 >> csclprd3-0-9 slots=16 max-slots=32 >> #total number of slots at this point is 104 >> #csclprd3-0-10 slots=16 max-slots=32 >> #csclprd3-0-11 slots=16 max-slots=32 >> #total number of slots at this point is 136 >> >> >> From: users [users-boun...@open-mpi.org <mailto:users-boun...@open-mpi.org>] >> on behalf of Ralph Castain [r...@open-mpi.org <mailto:r...@open-mpi.org>] >> Sent: Wednesday, April 08, 2015 11:31 AM >> To: Open MPI Users >> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >> >> Just for clarity: does the BIOS on the LGA2011 system have HT enabled? >> >>> On Apr 8, 2015, at 10:55 AM, Lane, William <william.l...@cshs.org >>> <mailto:william.l...@cshs.org>> wrote: >>> >>> Ralph, >>> >>> I added one of the newer LGA2011 nodes to my hostfile and >>> re-ran the benchmark successfully and saw some strange results WRT the >>> binding directives. Why are hyperthreading cores being used >>> on the LGA2011 system but not any of other systems which >>> are mostly hyperthreaded Westmeres)? Isn't the --use-hwthread-cpus >>> switch supposed to prevent OpenMPI from using hyperthreaded >>> cores? >>> >>> OpenMPI LAPACK invocation: >>> >>> $MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile >>> hostfile-single --mca btl_tcp_if_include eth0 --hetero-nodes >>> --use-hwthread-cpus --prefix $MPI_DIR $BENCH_DIR/$APP_DIR/$APP_BIN >>> >>> Where NSLOTS=72 >>> >>> hostfile: >>> csclprd3-6-1 slots=4 max-slots=4 >>> csclprd3-6-5 slots=4 max-slots=4 >>> csclprd3-0-0 slots=12 max-slots=24 >>> csclprd3-0-1 slots=6 max-slots=12 >>> csclprd3-0-2 slots=6 max-slots=12 >>> csclprd3-0-3 slots=6 max-slots=12 >>> csclprd3-0-4 slots=6 max-slots=12 >>> csclprd3-0-5 slots=6 max-slots=12 >>> csclprd3-0-6 slots=6 max-slots=12 >>> #total number of successfully tested non-hyperthreaded computes slots at >>> this point is 56 >>> csclprd3-0-7 slots=16 max-slots=32 >>> >>> LGA1366 Westmere w/two Intel Xeon X5675 6-core/12-hyperthread CPU's >>> >>> [csclprd3-0-0:11848] MCW rank 11 bound to socket 1[core 7[hwt 0]]: >>> [./././././.][./B/./././.] >>> [csclprd3-0-0:11848] MCW rank 12 bound to socket 0[core 2[hwt 0]]: >>> [././B/././.][./././././.] >>> [csclprd3-0-0:11848] MCW rank 13 bound to socket 1[core 8[hwt 0]]: >>> [./././././.][././B/././.] >>> [csclprd3-0-0:11848] MCW rank 14 bound to socket 0[core 3[hwt 0]]: >>> [./././B/./.][./././././.] >>> [csclprd3-0-0:11848] MCW rank 15 bound to socket 1[core 9[hwt 0]]: >>> [./././././.][./././B/./.] >>> [csclprd3-0-0:11848] MCW rank 16 bound to socket 0[core 4[hwt 0]]: >>> [././././B/.][./././././.] >>> [csclprd3-0-0:11848] MCW rank 17 bound to socket 1[core 10[hwt 0]]: >>> [./././././.][././././B/.] >>> [csclprd3-0-0:11848] MCW rank 18 bound to socket 0[core 5[hwt 0]]: >>> [./././././B][./././././.] >>> [csclprd3-0-0:11848] MCW rank 19 bound to socket 1[core 11[hwt 0]]: >>> [./././././.][./././././B] >>> [csclprd3-0-0:11848] MCW rank 8 bound to socket 0[core 0[hwt 0]]: >>> [B/././././.][./././././.] >>> [csclprd3-0-0:11848] MCW rank 9 bound to socket 1[core 6[hwt 0]]: >>> [./././././.][B/././././.] >>> [csclprd3-0-0:11848] MCW rank 10 bound to socket 0[core 1[hwt 0]]: >>> [./B/./././.][./././././.] >>> >>> but for the LGA2011 system w/two 8-core/16-hyperthread CPU's >>> >>> [csclprd3-0-7:30876] MCW rank 60 bound to socket 0[core 2[hwt 0-1]]: >>> [../../BB/../../../../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 61 bound to socket 1[core 10[hwt 0-1]]: >>> [../../../../../../../..][../../BB/../../../../..] >>> [csclprd3-0-7:30876] MCW rank 62 bound to socket 0[core 3[hwt 0-1]]: >>> [../../../BB/../../../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 63 bound to socket 1[core 11[hwt 0-1]]: >>> [../../../../../../../..][../../../BB/../../../..] >>> [csclprd3-0-7:30876] MCW rank 64 bound to socket 0[core 4[hwt 0-1]]: >>> [../../../../BB/../../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 65 bound to socket 1[core 12[hwt 0-1]]: >>> [../../../../../../../..][../../../../BB/../../..] >>> [csclprd3-0-7:30876] MCW rank 66 bound to socket 0[core 5[hwt 0-1]]: >>> [../../../../../BB/../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 67 bound to socket 1[core 13[hwt 0-1]]: >>> [../../../../../../../..][../../../../../BB/../..] >>> [csclprd3-0-7:30876] MCW rank 68 bound to socket 0[core 6[hwt 0-1]]: >>> [../../../../../../BB/..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 69 bound to socket 1[core 14[hwt 0-1]]: >>> [../../../../../../../..][../../../../../../BB/..] >>> [csclprd3-0-7:30876] MCW rank 70 bound to socket 0[core 7[hwt 0-1]]: >>> [../../../../../../../BB][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 71 bound to socket 1[core 15[hwt 0-1]]: >>> [../../../../../../../..][../../../../../../../BB] >>> [csclprd3-0-7:30876] MCW rank 56 bound to socket 0[core 0[hwt 0-1]]: >>> [BB/../../../../../../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 57 bound to socket 1[core 8[hwt 0-1]]: >>> [../../../../../../../..][BB/../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 58 bound to socket 0[core 1[hwt 0-1]]: >>> [../BB/../../../../../..][../../../../../../../..] >>> [csclprd3-0-7:30876] MCW rank 59 bound to socket 1[core 9[hwt 0-1]]: >>> [../../../../../../../..][../BB/../../../../../..] >>> >>> >>> >>> >>> From: users [users-boun...@open-mpi.org >>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>> Sent: Wednesday, April 08, 2015 10:26 AM >>> To: Open MPI Users >>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>> >>> >>>> On Apr 8, 2015, at 9:29 AM, Lane, William <william.l...@cshs.org >>>> <mailto:william.l...@cshs.org>> wrote: >>>> >>>> Ralph, >>>> >>>> Thanks for YOUR help, I never >>>> would've managed to get the LAPACK >>>> benchmark running on more than one >>>> node in our cluster without your help. >>>> >>>> Ralph, is hyperthreading more of a curse >>>> than an advantage for HPC applications? >>> >>> Wow, you’ll get a lot of argument over that issue! From what I can see, it >>> is very application dependent. Some apps appear to benefit, while others >>> can even suffer from it. >>> >>> I think we should support a mix of nodes in this usage, so I’ll try to come >>> up with a way to do so. >>> >>>> >>>> I'm going to go through all the OpenMPI >>>> articles on hyperthreading and NUMA to >>>> see if that will shed any light on these >>>> issues. >>>> >>>> -Bill L. >>>> >>>> >>>> From: users [users-boun...@open-mpi.org >>>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>>> Sent: Tuesday, April 07, 2015 7:32 PM >>>> To: Open MPI Users >>>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>>> >>>> I’m not sure our man pages are good enough to answer your question, but >>>> here is the URL >>>> >>>> http://www.open-mpi.org/doc/v1.8/ <http://www.open-mpi.org/doc/v1.8/> >>>> >>>> I’m a tad tied up right now, but I’ll try to address this prior to 1.8.5 >>>> release. Thanks for all that debug effort! Helps a bunch. >>>> >>>>> On Apr 7, 2015, at 1:17 PM, Lane, William <william.l...@cshs.org >>>>> <mailto:william.l...@cshs.org>> wrote: >>>>> >>>>> Ralph, >>>>> >>>>> I've finally had some luck using the following: >>>>> $MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile >>>>> hostfile-single --mca btl_tcp_if_include eth0 --hetero-nodes >>>>> --use-hwthread-cpus --prefix $MPI_DIR $BENCH_DIR/$APP_DIR/$APP_BIN >>>>> >>>>> Where $NSLOTS was 56 and my hostfile hostfile-single is: >>>>> >>>>> csclprd3-0-0 slots=12 max-slots=24 >>>>> csclprd3-0-1 slots=6 max-slots=12 >>>>> csclprd3-0-2 slots=6 max-slots=12 >>>>> csclprd3-0-3 slots=6 max-slots=12 >>>>> csclprd3-0-4 slots=6 max-slots=12 >>>>> csclprd3-0-5 slots=6 max-slots=12 >>>>> csclprd3-0-6 slots=6 max-slots=12 >>>>> csclprd3-6-1 slots=4 max-slots=4 >>>>> csclprd3-6-5 slots=4 max-slots=4 >>>>> >>>>> The max-slots differs from slots on some nodes >>>>> because I include the hyperthreaded cores in >>>>> the max-slots, the last two nodes have CPU's that >>>>> don't support hyperthreading at all. >>>>> >>>>> Does --use-hwthread-cpus prevent slots from >>>>> being assigned to hyperthreading cores? >>>>> >>>>> For some reason the manpage for OpenMPI 1.8.2 >>>>> isn't installed on our CentOS 6.3 systems is there a >>>>> URL I can I find a copy of the manpages for OpenMPI 1.8.2? >>>>> >>>>> Thanks for your help, >>>>> >>>>> -Bill Lane >>>>> >>>>> From: users [users-boun...@open-mpi.org >>>>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>>>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>>>> Sent: Monday, April 06, 2015 1:39 PM >>>>> To: Open MPI Users >>>>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>>>> >>>>> Hmmm…well, that shouldn’t be the issue. To check, try running it with >>>>> “bind-to none”. If you can get a backtrace telling us where it is >>>>> crashing, that would also help. >>>>> >>>>> >>>>>> On Apr 6, 2015, at 12:24 PM, Lane, William <william.l...@cshs.org >>>>>> <mailto:william.l...@cshs.org>> wrote: >>>>>> >>>>>> Ralph, >>>>>> >>>>>> For the following two different commandline invocations of the LAPACK >>>>>> benchmark >>>>>> >>>>>> $MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile >>>>>> hostfile-no_slots --mca btl_tcp_if_include eth0 --hetero-nodes >>>>>> --use-hwthread-cpus --bind-to hwthread --prefix $MPI_DIR >>>>>> $BENCH_DIR/$APP_DIR/$APP_BIN >>>>>> >>>>>> $MPI_DIR/bin/mpirun -np $NSLOTS --report-bindings --hostfile >>>>>> hostfile-no_slots --mca btl_tcp_if_include eth0 --hetero-nodes >>>>>> --bind-to-core --prefix $MPI_DIR $BENCH_DIR/$APP_DIR/$APP_BIN >>>>>> >>>>>> I'm receiving the same kinds of OpenMPI error messages (but for >>>>>> different nodes in the ring): >>>>>> >>>>>> [csclprd3-0-16:25940] *** Process received signal *** >>>>>> [csclprd3-0-16:25940] Signal: Bus error (7) >>>>>> [csclprd3-0-16:25940] Signal code: Non-existant physical address >>>>>> (2) >>>>>> [csclprd3-0-16:25940] Failing at address: 0x7f8b1b5a2600 >>>>>> >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> mpirun noticed that process rank 82 with PID 25936 on node >>>>>> csclprd3-0-16 exited on signal 7 (Bus error). >>>>>> >>>>>> -------------------------------------------------------------------------- >>>>>> 16 total processes killed (some possibly by mpirun during >>>>>> cleanup) >>>>>> >>>>>> It seems to occur on systems that have more than one, physical CPU >>>>>> installed. Could >>>>>> this be due to a lack of the correct NUMA libraries being installed? >>>>>> >>>>>> -Bill L. >>>>>> >>>>>> From: users [users-boun...@open-mpi.org >>>>>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>>>>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>>>>> Sent: Sunday, April 05, 2015 6:09 PM >>>>>> To: Open MPI Users >>>>>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>>>>> >>>>>> >>>>>>> On Apr 5, 2015, at 5:58 PM, Lane, William <william.l...@cshs.org >>>>>>> <mailto:william.l...@cshs.org>> wrote: >>>>>>> >>>>>>> I think some of the Intel Blade systems in the cluster are >>>>>>> dual core, but don't support hyperthreading. Maybe it >>>>>>> would be better to exclude hyperthreading altogether >>>>>>> from submitted OpenMPI jobs? >>>>>> >>>>>> Yes - or you can add "--hetero-nodes -use-hwthread-cpus --bind-to >>>>>> hwthread" to the cmd line. This tells mpirun that the nodes aren't all >>>>>> the same, and so it has to look at each node's topology instead of >>>>>> taking the first node as the template for everything. The second tells >>>>>> it to use the HTs as independent cpus where they are supported. >>>>>> >>>>>> I'm not entirely sure the suggestion will work - if we hit a place where >>>>>> HT isn't supported, we may balk at being asked to bind to HTs. I can >>>>>> probably make a change that supports this kind of hetero arrangement >>>>>> (perhaps something like bind-to pu) - might make it into 1.8.5 (we are >>>>>> just starting the release process on it now). >>>>>> >>>>>>> >>>>>>> OpenMPI doesn't crash, but it doesn't run the LAPACK >>>>>>> benchmark either. >>>>>>> >>>>>>> Thanks again Ralph. >>>>>>> >>>>>>> Bill L. >>>>>>> >>>>>>> From: users [users-boun...@open-mpi.org >>>>>>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>>>>>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>>>>>> Sent: Wednesday, April 01, 2015 8:40 AM >>>>>>> To: Open MPI Users >>>>>>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>>>>>> >>>>>>> Bingo - you said the magic word. This is a terminology issue. When we >>>>>>> say "core", we mean the old definition of "core", not "hyperthreads". >>>>>>> If you want to use HTs as your base processing unit and bind to them, >>>>>>> then you need to specify --bind-to hwthread. That warning should then >>>>>>> go away. >>>>>>> >>>>>>> We don't require a swap region be mounted - I didn't see anything in >>>>>>> your original message indicating that OMPI had actually crashed, but >>>>>>> just wasn't launching due to the above issue. Were you actually seeing >>>>>>> crashes as well? >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 1, 2015 at 8:31 AM, Lane, William <william.l...@cshs.org >>>>>>> <mailto:william.l...@cshs.org>> wrote: >>>>>>> Ralph, >>>>>>> >>>>>>> Here's the associated hostfile: >>>>>>> >>>>>>> #openMPI hostfile for csclprd3 >>>>>>> #max slots prevents oversubscribing csclprd3-0-9 >>>>>>> csclprd3-0-0 slots=12 max-slots=12 >>>>>>> csclprd3-0-1 slots=6 max-slots=6 >>>>>>> csclprd3-0-2 slots=6 max-slots=6 >>>>>>> csclprd3-0-3 slots=6 max-slots=6 >>>>>>> csclprd3-0-4 slots=6 max-slots=6 >>>>>>> csclprd3-0-5 slots=6 max-slots=6 >>>>>>> csclprd3-0-6 slots=6 max-slots=6 >>>>>>> csclprd3-0-7 slots=32 max-slots=32 >>>>>>> csclprd3-0-8 slots=32 max-slots=32 >>>>>>> csclprd3-0-9 slots=32 max-slots=32 >>>>>>> csclprd3-0-10 slots=32 max-slots=32 >>>>>>> csclprd3-0-11 slots=32 max-slots=32 >>>>>>> csclprd3-0-12 slots=12 max-slots=12 >>>>>>> csclprd3-0-13 slots=24 max-slots=24 >>>>>>> csclprd3-0-14 slots=16 max-slots=16 >>>>>>> csclprd3-0-15 slots=16 max-slots=16 >>>>>>> csclprd3-0-16 slots=24 max-slots=24 >>>>>>> csclprd3-0-17 slots=24 max-slots=24 >>>>>>> csclprd3-6-1 slots=4 max-slots=4 >>>>>>> csclprd3-6-5 slots=4 max-slots=4 >>>>>>> >>>>>>> The number of slots also includes hyperthreading >>>>>>> cores. >>>>>>> >>>>>>> One more question, would not having defined swap >>>>>>> partitions on all the nodes in the ring cause OpenMPI >>>>>>> to crash? Because no swap partitions are defined >>>>>>> for any of the above systems. >>>>>>> >>>>>>> -Bill L. >>>>>>> >>>>>>> >>>>>>> From: users [users-boun...@open-mpi.org >>>>>>> <mailto:users-boun...@open-mpi.org>] on behalf of Ralph Castain >>>>>>> [r...@open-mpi.org <mailto:r...@open-mpi.org>] >>>>>>> Sent: Wednesday, April 01, 2015 5:04 AM >>>>>>> To: Open MPI Users >>>>>>> Subject: Re: [OMPI users] OpenMPI 1.8.2 problems on CentOS 6.3 >>>>>>> >>>>>>> The warning about binding to memory is due to not having numactl-devel >>>>>>> installed on the system. The job would still run, but we are warning >>>>>>> you that we cannot bind memory to the same domain as the core where we >>>>>>> bind the process. Can cause poor performance, but not fatal. I forget >>>>>>> the name of the param, but you can tell us to "shut up" :-) >>>>>>> >>>>>>> The other warning/error indicates that we aren't seeing enough cores on >>>>>>> the allocation you gave us via the hostile to support one proc/core - >>>>>>> i.e., we didn't at least 128 cores in the sum of the nodes you told us >>>>>>> about. I take it you were expecting that there were that many or more? >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>> >>>>>>> On Wed, Apr 1, 2015 at 12:54 AM, Lane, William <william.l...@cshs.org >>>>>>> <mailto:william.l...@cshs.org>> wrote: >>>>>>> I'm having problems running OpenMPI jobs >>>>>>> (using a hostfile) on an HPC cluster running >>>>>>> ROCKS on CentOS 6.3. I'm running OpenMPI >>>>>>> outside of Sun Grid Engine (i.e. it is not submitted >>>>>>> as a job to SGE). The program being run is a LAPACK >>>>>>> benchmark. The commandline parameter I'm >>>>>>> using to run the jobs is: >>>>>>> >>>>>>> $MPI_DIR/bin/mpirun -np $NSLOTS -bind-to-core -report-bindings >>>>>>> --hostfile hostfile --mca btl_tcp_if_include eth0 --prefix $MPI_DIR >>>>>>> $BENCH_DIR/$APP_DIR/$APP_BIN >>>>>>> >>>>>>> Where MPI_DIR=/hpc/apps/mpi/openmpi/1.8.2/ >>>>>>> NSLOTS=128 >>>>>>> >>>>>>> I'm getting errors of the form and OpenMPI never runs the LAPACK >>>>>>> benchmark: >>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------------- >>>>>>> WARNING: a request was made to bind a process. While the system >>>>>>> supports binding the process itself, at least one node does NOT >>>>>>> support binding memory to the process location. >>>>>>> >>>>>>> Node: csclprd3-0-11 >>>>>>> >>>>>>> This usually is due to not having the required NUMA support installed >>>>>>> on the node. In some Linux distributions, the required support is >>>>>>> contained in the libnumactl and libnumactl-devel packages. >>>>>>> This is a warning only; your job will continue, though performance >>>>>>> may be degraded. >>>>>>> >>>>>>> -------------------------------------------------------------------------- >>>>>>> >>>>>>> >>>>>>> -------------------------------------------------------------------------- >>>>>>> A request was made to bind to that would result in binding more >>>>>>> processes than cpus on a resource: >>>>>>> >>>>>>> Bind to: CORE >>>>>>> Node: csclprd3-0-11 >>>>>>> #processes: 2 >>>>>>> #cpus: 1 >>>>>>> >>>>>>> You can override this protection by adding the "overload-allowed" >>>>>>> option to your binding directive. >>>>>>> >>>>>>> -------------------------------------------------------------------------- >>>>>>> >>>>>>> The only installed numa packages are: >>>>>>> numactl.x86_64 >>>>>>> 2.0.7-3.el6 @centos6.3-x86_64-0/$ >>>>>>> >>>>>>> When I search for the available NUMA packages I find: >>>>>>> >>>>>>> yum search numa | less >>>>>>> >>>>>>> Loaded plugins: fastestmirror >>>>>>> Loading mirror speeds from cached hostfile >>>>>>> ============================== N/S Matched: numa >>>>>>> =============================== >>>>>>> numactl-devel.i686 : Development package for building >>>>>>> Applications that use numa >>>>>>> numactl-devel.x86_64 : Development package for building >>>>>>> Applications that use >>>>>>> : numa >>>>>>> numad.x86_64 : NUMA user daemon >>>>>>> numactl.i686 : Library for tuning for Non Uniform Memory Access >>>>>>> machines >>>>>>> numactl.x86_64 : Library for tuning for Non Uniform Memory >>>>>>> Access machines >>>>>>> >>>>>>> Do I need to install additional and/or different NUMA packages in order >>>>>>> to get OpenMPI to work >>>>>>> on this cluster? >>>>>>> >>>>>>> -Bill Lane >>>>>>> IMPORTANT WARNING: This message is intended for the use of the person >>>>>>> or entity to which it is addressed and may contain information that is >>>>>>> privileged and confidential, the disclosure of which is governed by >>>>>>> applicable law. If the reader of this message is not the intended >>>>>>> recipient, or the employee or agent responsible for delivering it to >>>>>>> the intended recipient, you are hereby notified that any dissemination, >>>>>>> distribution or copying of this information is strictly prohibited. >>>>>>> Thank you for your cooperation. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>>> Searchable archives: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/index.php >>>>>>> <http://www.open-mpi.org/community/lists/users/2015/04/index.php> >>>>>>> >>>>>>> IMPORTANT WARNING: This message is intended for the use of the person >>>>>>> or entity to which it is addressed and may contain information that is >>>>>>> privileged and confidential, the disclosure of which is governed by >>>>>>> applicable law. If the reader of this message is not the intended >>>>>>> recipient, or the employee or agent responsible for delivering it to >>>>>>> the intended recipient, you are hereby notified that any dissemination, >>>>>>> distribution or copying of this information is strictly prohibited. >>>>>>> Thank you for your cooperation. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26589.php >>>>>>> <http://www.open-mpi.org/community/lists/users/2015/04/26589.php> >>>>>>> >>>>>>> IMPORTANT WARNING: This message is intended for the use of the person >>>>>>> or entity to which it is addressed and may contain information that is >>>>>>> privileged and confidential, the disclosure of which is governed by >>>>>>> applicable law. If the reader of this message is not the intended >>>>>>> recipient, or the employee or agent responsible for delivering it to >>>>>>> the intended recipient, you are hereby notified that any dissemination, >>>>>>> distribution or copying of this information is strictly prohibited. >>>>>>> Thank you for your cooperation. >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26611.php >>>>>>> <http://www.open-mpi.org/community/lists/users/2015/04/26611.php> >>>>>> IMPORTANT WARNING: This message is intended for the use of the person or >>>>>> entity to which it is addressed and may contain information that is >>>>>> privileged and confidential, the disclosure of which is governed by >>>>>> applicable law. If the reader of this message is not the intended >>>>>> recipient, or the employee or agent responsible for delivering it to the >>>>>> intended recipient, you are hereby notified that any dissemination, >>>>>> distribution or copying of this information is strictly prohibited. >>>>>> Thank you for your cooperation. >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2015/04/26618.php >>>>>> <http://www.open-mpi.org/community/lists/users/2015/04/26618.php> >>>>> IMPORTANT WARNING: This message is intended for the use of the person or >>>>> entity to which it is addressed and may contain information that is >>>>> privileged and confidential, the disclosure of which is governed by >>>>> applicable law. If the reader of this message is not the intended >>>>> recipient, or the employee or agent responsible for delivering it to the >>>>> intended recipient, you are hereby notified that any dissemination, >>>>> distribution or copying of this information is strictly prohibited. Thank >>>>> you for your cooperation. _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2015/04/26643.php >>>>> <http://www.open-mpi.org/community/lists/users/2015/04/26643.php> >>>> IMPORTANT WARNING: This message is intended for the use of the person or >>>> entity to which it is addressed and may contain information that is >>>> privileged and confidential, the disclosure of which is governed by >>>> applicable law. If the reader of this message is not the intended >>>> recipient, or the employee or agent responsible for delivering it to the >>>> intended recipient, you are hereby notified that any dissemination, >>>> distribution or copying of this information is strictly prohibited. Thank >>>> you for your cooperation. _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2015/04/26655.php >>>> <http://www.open-mpi.org/community/lists/users/2015/04/26655.php> >>> IMPORTANT WARNING: This message is intended for the use of the person or >>> entity to which it is addressed and may contain information that is >>> privileged and confidential, the disclosure of which is governed by >>> applicable law. If the reader of this message is not the intended >>> recipient, or the employee or agent responsible for delivering it to the >>> intended recipient, you are hereby notified that any dissemination, >>> distribution or copying of this information is strictly prohibited. Thank >>> you for your cooperation. _______________________________________________ >>> users mailing list >>> us...@open-mpi.org <mailto:us...@open-mpi.org> >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2015/04/26659.php >>> <http://www.open-mpi.org/community/lists/users/2015/04/26659.php> >> IMPORTANT WARNING: This message is intended for the use of the person or >> entity to which it is addressed and may contain information that is >> privileged and confidential, the disclosure of which is governed by >> applicable law. If the reader of this message is not the intended recipient, >> or the employee or agent responsible for delivering it to the intended >> recipient, you are hereby notified that any dissemination, distribution or >> copying of this information is strictly prohibited. Thank you for your >> cooperation. _______________________________________________ >> users mailing list >> us...@open-mpi.org <mailto:us...@open-mpi.org> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> <http://www.open-mpi.org/mailman/listinfo.cgi/users> >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2015/04/26664.php >> <http://www.open-mpi.org/community/lists/users/2015/04/26664.php>
