Ack - that was my fault. Too early on a monday morning. This seems to
work perfectly when I correctly submit a job! Thanks!
Dan
On Mon, Jun 9, 2014 at 9:34 AM, Dan Dietz <ddi...@purdue.edu> wrote:
> Yes, you're exactly right - this system has 2 Phi cards per node. I
> believe the "PCI 8086" device in the lstopo output is them. Possibly
> related, we've observed a weird bug with Torque and the allocation it
> provides when you request the Phis. When requesting them you get a
> nodefile with only 1 entry per node (instead of 16). This doesn't seem
> to be a problem when you ignore the phis in the request, which I did
> in these tests. Not sure if that's related.
>
> This system is down for maintenance for a few days so it'll be a bit
> before I can get more out of it.
>
> On the slot issue - OK, which machinefile should I use? If I use my
> modified one (2 lines per node) I still get the same error when using
> "socket" or "slot".
>
> When using the Torque provided one I seem to get some weird behavior.
> With 2 processes, I end up with 1 on each of my allocated nodes and
> then I get this:
>
> $ mpirun --display-allocation -np 2 -map-by socket:pe=8 ./hello
>
> ====================== ALLOCATED NODES ======================
> carter-a247: slots=8 max_slots=0 slots_inuse=0
> carter-a521: slots=8 max_slots=0 slots_inuse=0
> =================================================================
> ...
> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,4,6,7}
> OMP: Info #156: KMP_AFFINITY: 4 available OS procs
> OMP: Info #157: KMP_AFFINITY: Uniform topology
> OMP: Info #179: KMP_AFFINITY: 1 packages x 4 cores/pkg x 1
> threads/core (4 total cores)
> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set {0,4,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set {0,4,6,7}
> ...
> OMP: Info #204: KMP_AFFINITY: decoding x2APIC ids.
> OMP: Info #202: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11
> info
> OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: {0,1,2,3,4,5,6,7}
> OMP: Info #156: KMP_AFFINITY: 8 available OS procs
> OMP: Info #157: KMP_AFFINITY: Uniform topology
> OMP: Info #179: KMP_AFFINITY: 1 packages x 8 cores/pkg x 1
> threads/core (8 total cores)
> OMP: Info #147: KMP_AFFINITY: Internal thread 0 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 1 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 2 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 3 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 4 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 5 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 6 bound to OS proc set
> {0,1,2,3,4,5,6,7}
> OMP: Info #147: KMP_AFFINITY: Internal thread 7 bound to OS proc set
> {0,1,2,3,4,5,6,7}
>
>
> Any ideas?
>
> Thanks!
> Dan
>
>
>
> On Sun, Jun 8, 2014 at 4:54 PM, Ralph Castain <r...@open-mpi.org> wrote:
>> I'm having no luck poking at this segfault issue. For some strange reason,
>> we seem to think there are coprocessors on those remote nodes - e.g., a Phi
>> card. Yet your lstopo output doesn't seem to show it.
>>
>> Out of curiosity, can you try running this with "-mca plm rsh"? This will
>> substitute the rsh/ssh launcher in place of Torque - assuming your system
>> will allow it, this will let me see if the problem is somewhere in the
>> Torque launcher or elsewhere in OMPI.
>>
>> Thanks
>> Ralph
>>
>> On Jun 6, 2014, at 12:48 PM, Dan Dietz <ddi...@purdue.edu> wrote:
>>
>>> No problem -
>>>
>>> These are model name : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz chips.
>>> 2 per node, 8 cores each. No threading enabled.
>>>
>>> $ lstopo
>>> Machine (64GB)
>>> NUMANode L#0 (P#0 32GB)
>>> Socket L#0 + L3 L#0 (20MB)
>>> L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0
>>> (P#0)
>>> L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1
>>> (P#1)
>>> L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2
>>> (P#2)
>>> L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3
>>> (P#3)
>>> L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4
>>> (P#4)
>>> L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5
>>> (P#5)
>>> L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6
>>> (P#6)
>>> L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7
>>> (P#7)
>>> HostBridge L#0
>>> PCIBridge
>>> PCI 1000:0087
>>> Block L#0 "sda"
>>> PCIBridge
>>> PCI 8086:2250
>>> PCIBridge
>>> PCI 8086:1521
>>> Net L#1 "eth0"
>>> PCI 8086:1521
>>> Net L#2 "eth1"
>>> PCIBridge
>>> PCI 102b:0533
>>> PCI 8086:1d02
>>> NUMANode L#1 (P#1 32GB)
>>> Socket L#1 + L3 L#1 (20MB)
>>> L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8
>>> (P#8)
>>> L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9
>>> (P#9)
>>> L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
>>> + PU L#10 (P#10)
>>> L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
>>> + PU L#11 (P#11)
>>> L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
>>> + PU L#12 (P#12)
>>> L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
>>> + PU L#13 (P#13)
>>> L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
>>> + PU L#14 (P#14)
>>> L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
>>> + PU L#15 (P#15)
>>> HostBridge L#5
>>> PCIBridge
>>> PCI 15b3:1011
>>> Net L#3 "ib0"
>>> OpenFabrics L#4 "mlx5_0"
>>> PCIBridge
>>> PCI 8086:2250
>>>
>>> From the segfault below. I tried reproducing the crash on less than an
>>> 4 node allocation but wasn't able to.
>>>
>>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ mpirun -np 2
>>> -machinefile ./nodes -mca plm_base_verbose 10 ./hello
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> registering plm components
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> found loaded component isolated
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> component isolated has no register or open function
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> found loaded component slurm
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> component slurm register function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> found loaded component rsh
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> component rsh register function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> found loaded component tm
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_register:
>>> component tm register function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: opening
>>> plm components
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>>> loaded component isolated
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>>> component isolated open function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>>> loaded component slurm
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>>> component slurm open function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>>> loaded component rsh
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>>> component rsh open function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open: found
>>> loaded component tm
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: components_open:
>>> component tm open function successful
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select: Auto-selecting plm
>>> components
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Querying
>>> component [isolated]
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Query of
>>> component [isolated] set priority to 0
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Querying
>>> component [slurm]
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Skipping
>>> component [slurm]. Query failed to return a module
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Querying
>>> component [rsh]
>>> [conte-a009.rcac.purdue.edu:55685] [[INVALID],INVALID] plm:rsh_lookup
>>> on agent ssh : rsh path NULL
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Query of
>>> component [rsh] set priority to 10
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Querying
>>> component [tm]
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Query of
>>> component [tm] set priority to 75
>>> [conte-a009.rcac.purdue.edu:55685] mca:base:select:( plm) Selected
>>> component [tm]
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: component isolated
>>> closed
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: unloading
>>> component isolated
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: component slurm closed
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: unloading component
>>> slurm
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: component rsh closed
>>> [conte-a009.rcac.purdue.edu:55685] mca: base: close: unloading component rsh
>>> [conte-a009.rcac.purdue.edu:55685] plm:base:set_hnp_name: initial bias
>>> 55685 nodename hash 3965217721
>>> [conte-a009.rcac.purdue.edu:55685] plm:base:set_hnp_name: final jobfam 24164
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:receive start comm
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:setup_job
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:setup_vm
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:setup_vm creating
>>> map
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:setup_vm add
>>> new daemon [[24164,0],1]
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:base:setup_vm
>>> assigning new daemon [[24164,0],1] to node conte-a055
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: launching vm
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: final top-level
>>> argv:
>>> orted -mca ess tm -mca orte_ess_jobid 1583611904 -mca orte_ess_vpid
>>> <template> -mca orte_ess_num_procs 2 -mca orte_hnp_uri
>>> "1583611904.0;tcp://172.18.96.49,172.31.1.254,172.31.2.254,172.18.112.49:37380"
>>> -mca plm_base_verbose 10
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: resetting
>>> LD_LIBRARY_PATH:
>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib:/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib:/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib:/usr/pbs/lib:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/compiler/lib/intel64:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/mpirt/lib/intel64:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/ipp/lib/intel64:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/mkl/lib/intel64:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/tbb/lib/intel64:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/mic/myo/lib:/apps/rhel6/intel/opencl-1.2-3.2.1.16712/lib64
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: resetting
>>> PATH:
>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin:/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin:/scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin:/apps/rhel6/intel/composer_xe_2013_sp1.2.144/bin/intel64:/opt/intel/mic/bin:/apps/rhel6/intel/inspector_xe_2013/bin64:/apps/rhel6/intel/advisor_xe_2013/bin64:/apps/rhel6/intel/vtune_amplifier_xe_2013/bin64:/apps/rhel6/intel/opencl-1.2-3.2.1.16712/bin:/usr/lib64/qt-3.3/bin:/opt/moab/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/hpss/bin:/opt/hsi/bin:/opt/ibutils/bin:/usr/pbs/bin:/opt/moab/bin:/usr/site/rcac/scripts:/usr/site/rcac/support_scripts:/usr/site/rcac/bin:/usr/site/rcac/sbin:/usr/sbin
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: launching on
>>> node conte-a055
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm: executing:
>>> orted -mca ess tm -mca orte_ess_jobid 1583611904 -mca orte_ess_vpid 1
>>> -mca orte_ess_num_procs 2 -mca orte_hnp_uri
>>> "1583611904.0;tcp://172.18.96.49,172.31.1.254,172.31.2.254,172.18.112.49:37380"
>>> -mca plm_base_verbose 10
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] plm:tm:launch:
>>> finished spawning orteds
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_register:
>>> registering plm components
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_register:
>>> found loaded component rsh
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_register:
>>> component rsh register function successful
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_open: opening
>>> plm components
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_open: found
>>> loaded component rsh
>>> [conte-a055.rcac.purdue.edu:32094] mca: base: components_open:
>>> component rsh open function successful
>>> [conte-a055.rcac.purdue.edu:32094] mca:base:select: Auto-selecting plm
>>> components
>>> [conte-a055.rcac.purdue.edu:32094] mca:base:select:( plm) Querying
>>> component [rsh]
>>> [conte-a055.rcac.purdue.edu:32094] [[24164,0],1] plm:rsh_lookup on
>>> agent ssh : rsh path NULL
>>> [conte-a055.rcac.purdue.edu:32094] mca:base:select:( plm) Query of
>>> component [rsh] set priority to 10
>>> [conte-a055.rcac.purdue.edu:32094] mca:base:select:( plm) Selected
>>> component [rsh]
>>> [conte-a055.rcac.purdue.edu:32094] [[24164,0],1] plm:rsh_setup on
>>> agent ssh : rsh path NULL
>>> [conte-a055.rcac.purdue.edu:32094] [[24164,0],1] plm:base:receive start comm
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0]
>>> plm:base:orted_report_launch from daemon [[24164,0],1]
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0]
>>> plm:base:orted_report_launch from daemon [[24164,0],1] on node
>>> conte-a055
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] RECEIVED TOPOLOGY
>>> FROM NODE conte-a055
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0] NEW TOPOLOGY - ADDING
>>> [conte-a009.rcac.purdue.edu:55685] [[24164,0],0]
>>> plm:base:orted_report_launch completed for daemon [[24164,0],1] at
>>> contact
>>> 1583611904.1;tcp://172.18.96.95,172.31.1.254,172.31.2.254,172.18.112.95:58312
>>> [conte-a009:55685] *** Process received signal ***
>>> [conte-a009:55685] Signal: Segmentation fault (11)
>>> [conte-a009:55685] Signal code: Address not mapped (1)
>>> [conte-a009:55685] Failing at address: 0x4c
>>> [conte-a009:55685] [ 0] /lib64/libpthread.so.0[0x327f80f500]
>>> [conte-a009:55685] [ 1]
>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib/libopen-rte.so.7(orte_plm_base_complete_setup+0x951)[0x2b5b069a50e1]
>>> [conte-a009:55685] [ 2]
>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib/libopen-pal.so.6(opal_libevent2021_event_base_loop+0xa05)[0x2b5b075ff145]
>>> [conte-a009:55685] [ 3] mpirun(orterun+0x1ffd)[0x4073b5]
>>> [conte-a009:55685] [ 4] mpirun(main+0x20)[0x4048f4]
>>> [conte-a009:55685] [ 5]
>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x327f41ecdd]
>>> [conte-a009:55685] [ 6] mpirun[0x404819]
>>> [conte-a009:55685] *** End of error message ***
>>> Segmentation fault (core dumped)
>>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$
>>> [conte-a055.rcac.purdue.edu:32094] [[24164,0],1] plm:base:receive stop
>>> comm
>>>
>>> On Fri, Jun 6, 2014 at 3:00 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>> Sorry to pester with questions, but I'm trying to narrow down the issue.
>>>>
>>>> * What kind of chips are on these machines?
>>>>
>>>> * If they have h/w threads, are they enabled?
>>>>
>>>> * you might have lstopo on one of those machines - could you pass along
>>>> its output? Otherwise, you can run a simple "mpirun -n 1 -mca
>>>> ess_base_verbose 20 hostname" and it will print out. Only need one node in
>>>> your allocation as we don't need a fountain of output.
>>>>
>>>> I'll look into the segfault - hard to understand offhand, but could be an
>>>> uninitialized variable. If you have a chance, could you rerun that test
>>>> with "-mca plm_base_verbose 10" on the cmd line?
>>>>
>>>> Thanks again
>>>> Ralph
>>>>
>>>> On Jun 6, 2014, at 10:31 AM, Dan Dietz <ddi...@purdue.edu> wrote:
>>>>
>>>>> Thanks for the reply. I tried out the --display-allocation option with
>>>>> several different combinations and have attached the output. I see
>>>>> this behavior on both RHEL6.4, RHEL6.5, and RHEL5.10 clusters.
>>>>>
>>>>>
>>>>> Here's debugging info on the segfault. Does that help? FWIW this does
>>>>> not seem to crash on the RHEL5 cluster or RHEL6.5 cluster. Just
>>>>> crashes on RHEL6.4.
>>>>>
>>>>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ gdb -c core.22623
>>>>> `which mpirun`
>>>>> No symbol table is loaded. Use the "file" command.
>>>>> GNU gdb (GDB) 7.5-1.3.187
>>>>> Copyright (C) 2012 Free Software Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later
>>>>> <http://gnu.org/licenses/gpl.html>
>>>>> This is free software: you are free to change and redistribute it.
>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as "x86_64-unknown-linux-gnu".
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/>...
>>>>> Reading symbols from
>>>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/bin/mpirun...done.
>>>>> [New LWP 22623]
>>>>> [New LWP 22624]
>>>>>
>>>>> warning: Can't read pathname for load map: Input/output error.
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>>> Core was generated by `mpirun -np 2 -machinefile ./nodes ./hello'.
>>>>> Program terminated with signal 11, Segmentation fault.
>>>>> #0 0x00002acc602920e1 in orte_plm_base_complete_setup (fd=-1,
>>>>> args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
>>>>> 422 node->hostid = node->daemon->name.vpid;
>>>>> (gdb) bt
>>>>> #0 0x00002acc602920e1 in orte_plm_base_complete_setup (fd=-1,
>>>>> args=-1, cbdata=0x20c0840) at base/plm_base_launch_support.c:422
>>>>> #1 0x00002acc60eec145 in opal_libevent2021_event_base_loop () from
>>>>> /scratch/conte/d/ddietz/openmpi-1.8.1-debug/intel-14.0.2.144/lib/libopen-pal.so.6
>>>>> #2 0x00000000004073b5 in orterun (argc=6, argv=0x7fff5bb2a3a8) at
>>>>> orterun.c:1077
>>>>> #3 0x00000000004048f4 in main (argc=6, argv=0x7fff5bb2a3a8) at main.c:13
>>>>>
>>>>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ cat nodes
>>>>> conte-a009
>>>>> conte-a009
>>>>> conte-a055
>>>>> conte-a055
>>>>> ddietz@conte-a009:/scratch/conte/d/ddietz/hello$ uname -r
>>>>> 2.6.32-358.14.1.el6.x86_64
>>>>>
>>>>> On Thu, Jun 5, 2014 at 7:54 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>>>
>>>>>> On Jun 5, 2014, at 2:13 PM, Dan Dietz <ddi...@purdue.edu> wrote:
>>>>>>
>>>>>>> Hello all,
>>>>>>>
>>>>>>> I'd like to bind 8 cores to a single MPI rank for hybrid MPI/OpenMP
>>>>>>> codes. In OMPI 1.6.3, I can do:
>>>>>>>
>>>>>>> $ mpirun -np 2 -cpus-per-rank 8 -machinefile ./nodes ./hello
>>>>>>>
>>>>>>> I get one rank bound to procs 0-7 and the other bound to 8-15. Great!
>>>>>>>
>>>>>>> But I'm having some difficulties doing this with openmpi 1.8.1:
>>>>>>>
>>>>>>> $ mpirun -np 2 -cpus-per-rank 8 -machinefile ./nodes ./hello
>>>>>>> --------------------------------------------------------------------------
>>>>>>> The following command line options and corresponding MCA parameter have
>>>>>>> been deprecated and replaced as follows:
>>>>>>>
>>>>>>> Command line options:
>>>>>>> Deprecated: --cpus-per-proc, -cpus-per-proc, --cpus-per-rank,
>>>>>>> -cpus-per-rank
>>>>>>> Replacement: --map-by <obj>:PE=N
>>>>>>>
>>>>>>> Equivalent MCA parameter:
>>>>>>> Deprecated: rmaps_base_cpus_per_proc
>>>>>>> Replacement: rmaps_base_mapping_policy=<obj>:PE=N
>>>>>>>
>>>>>>> The deprecated forms *will* disappear in a future version of Open MPI.
>>>>>>> Please update to the new syntax.
>>>>>>> --------------------------------------------------------------------------
>>>>>>> --------------------------------------------------------------------------
>>>>>>> There are not enough slots available in the system to satisfy the 2
>>>>>>> slots
>>>>>>> that were requested by the application:
>>>>>>> ./hello
>>>>>>>
>>>>>>> Either request fewer slots for your application, or make more slots
>>>>>>> available
>>>>>>> for use.
>>>>>>> --------------------------------------------------------------------------
>>>>>>>
>>>>>>> OK, let me try the new syntax...
>>>>>>>
>>>>>>> $ mpirun -np 2 --map-by core:pe=8 -machinefile ./nodes ./hello
>>>>>>> --------------------------------------------------------------------------
>>>>>>> There are not enough slots available in the system to satisfy the 2
>>>>>>> slots
>>>>>>> that were requested by the application:
>>>>>>> ./hello
>>>>>>>
>>>>>>> Either request fewer slots for your application, or make more slots
>>>>>>> available
>>>>>>> for use.
>>>>>>> --------------------------------------------------------------------------
>>>>>>>
>>>>>>> What am I doing wrong? The documentation on these new options is
>>>>>>> somewhat poor and confusing so I'm probably doing something wrong. If
>>>>>>> anyone could provide some pointers here it'd be much appreciated! If
>>>>>>> it's not something simple and you need config logs and such please let
>>>>>>> me know.
>>>>>>
>>>>>> Looks like we think there are less than 16 slots allocated on that node.
>>>>>> What is in this "nodes" file? Without it, OMPI should read the Torque
>>>>>> allocation directly. You might check what we think the allocation is by
>>>>>> adding --display-allocation to the cmd line
>>>>>>
>>>>>>>
>>>>>>> As a side note -
>>>>>>>
>>>>>>> If I try this using the PBS nodefile with the above, I get a confusing
>>>>>>> message:
>>>>>>>
>>>>>>> --------------------------------------------------------------------------
>>>>>>> A request for multiple cpus-per-proc was given, but a directive
>>>>>>> was also give to map to an object level that has less cpus than
>>>>>>> requested ones:
>>>>>>>
>>>>>>> #cpus-per-proc: 8
>>>>>>> number of cpus: 1
>>>>>>> map-by: BYCORE:NOOVERSUBSCRIBE
>>>>>>>
>>>>>>> Please specify a mapping level that has more cpus, or else let us
>>>>>>> define a default mapping that will allow multiple cpus-per-proc.
>>>>>>> --------------------------------------------------------------------------
>>>>>>>
>>>>>>> From what I've gathered this is because I have a node listed 16 times
>>>>>>> in my PBS nodefile so it's assuming then I have 1 core per node?
>>>>>>
>>>>>>
>>>>>> No - if listed 16 times, it should compute 16 slots. Try adding
>>>>>> --display-allocation to your cmd line and it should tell you how many
>>>>>> slots are present.
>>>>>>
>>>>>> However, it doesn't assume there is a core for each slot. Instead, it
>>>>>> detects the cores directly on the node. It sounds like it isn't seeing
>>>>>> them for some reason. What OS are you running on that node?
>>>>>>
>>>>>> FWIW: the 1.6 series has a different detection system for cores. Could
>>>>>> be something is causing problems for the new one.
>>>>>>
>>>>>>> Some
>>>>>>> better documentation here would be helpful. I haven't been able to
>>>>>>> figure out how to use the "oversubscribe" option listed in the docs.
>>>>>>> Not that I really want to oversubscribe, of course, I need to modify
>>>>>>> the nodefile, but this just stumped me for a while as 1.6.3 didn't
>>>>>>> have this behavior.
>>>>>>>
>>>>>>>
>>>>>>> As a extra bonus, I get a segfault in this situation:
>>>>>>>
>>>>>>> $ mpirun -np 2 -machinefile ./nodes ./hello
>>>>>>> [conte-a497:13255] *** Process received signal ***
>>>>>>> [conte-a497:13255] Signal: Segmentation fault (11)
>>>>>>> [conte-a497:13255] Signal code: Address not mapped (1)
>>>>>>> [conte-a497:13255] Failing at address: 0x2c
>>>>>>> [conte-a497:13255] [ 0] /lib64/libpthread.so.0[0x3c9460f500]
>>>>>>> [conte-a497:13255] [ 1]
>>>>>>> /apps/rhel6/openmpi/1.8.1/intel-14.0.2.144/lib/libopen-rte.so.7(orte_plm_base_complete_setup+0x615)[0x2ba960a59015]
>>>>>>> [conte-a497:13255] [ 2]
>>>>>>> /apps/rhel6/openmpi/1.8.1/intel-14.0.2.144/lib/libopen-pal.so.6(opal_libevent2021_event_base_loop+0xa05)[0x2ba961666715]
>>>>>>> [conte-a497:13255] [ 3] mpirun(orterun+0x1b45)[0x40684f]
>>>>>>> [conte-a497:13255] [ 4] mpirun(main+0x20)[0x4047f4]
>>>>>>> [conte-a497:13255] [ 5]
>>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x3a1bc1ecdd]
>>>>>>> [conte-a497:13255] [ 6] mpirun[0x404719]
>>>>>>> [conte-a497:13255] *** End of error message ***
>>>>>>> Segmentation fault (core dumped)
>>>>>>>
>>>>>>
>>>>>> Huh - that's odd. Could you perhaps configure OMPI with --enable-debug
>>>>>> and gdb the core file to tell us the line numbers involved?
>>>>>>
>>>>>> Thanks
>>>>>> Ralph
>>>>>>
>>>>>>>
>>>>>>> My "nodes" file simply contains the first two lines of my original
>>>>>>> $PBS_NODEFILE provided by Torque. See above why I modified. Works fine
>>>>>>> if use the full file.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks in advance for any pointers you all may have!
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dan Dietz
>>>>>>> Scientific Applications Analyst
>>>>>>> ITaP Research Computing, Purdue University
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dan Dietz
>>>>> Scientific Applications Analyst
>>>>> ITaP Research Computing, Purdue University
>>>>> <slots>_______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>>
>>> --
>>> Dan Dietz
>>> Scientific Applications Analyst
>>> ITaP Research Computing, Purdue University
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> --
> Dan Dietz
> Scientific Applications Analyst
> ITaP Research Computing, Purdue University
--
Dan Dietz
Scientific Applications Analyst
ITaP Research Computing, Purdue University
