On Nov 13, 2013, at 4:43 PM, tmish...@jcity.maeda.co.jp wrote:
>
>
> Yes, the node08 has 8 slots but the process I run is also 8.
>
> #PBS -l nodes=node08:ppn=8
>
> Therefore, I think it should allow this allocation. Is that right?
Correct
>
> My question is why scritp1 works and script2 does not. They are
> almost same.
>
> #PBS -l nodes=node08:ppn=8
> export OMP_NUM_THREADS=1
> cd $PBS_O_WORKDIR
> cp $PBS_NODEFILE pbs_hosts
> NPROCS=`wc -l < pbs_hosts`
>
> #SCRITP1
> mpirun -report-bindings -bind-to core Myprog
>
> #SCRIPT2
> mpirun -machinefile pbs_hosts -np ${NPROCS} -report-bindings -bind-to core
This version is not only reading the PBS allocation, but also invoking the
hostfile filter on top of it. Different code path. I'll take a look - it should
still match up assuming NPROCS=8. Any possibility that it is a different
number? I don't recall, but isn't there some extra lines in the nodefile -
e.g., comments?
> Myprog
>
> tmishima
>
>> I guess here's my confusion. If you are using only one node, and that
> node has 8 allocated slots, then we will not allow you to run more than 8
> processes on that node unless you specifically provide
>> the --oversubscribe flag. This is because you are operating in a managed
> environment (in this case, under Torque), and so we treat the allocation as
> "mandatory" by default.
>>
>> I suspect that is the issue here, in which case the system is behaving as
> it should.
>>
>> Is the above accurate?
>>
>>
>> On Nov 13, 2013, at 4:11 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>> It has nothing to do with LAMA as you aren't using that mapper.
>>>
>>> How many nodes are in this allocation?
>>>
>>> On Nov 13, 2013, at 4:06 PM, tmish...@jcity.maeda.co.jp wrote:
>>>
>>>>
>>>>
>>>> Hi Ralph, this is an additional information.
>>>>
>>>> Here is the main part of output by adding "-mca rmaps_base_verbose
> 50".
>>>>
>>>> [node08.cluster:26952] [[56581,0],0] plm:base:setup_vm
>>>> [node08.cluster:26952] [[56581,0],0] plm:base:setup_vm creating map
>>>> [node08.cluster:26952] [[56581,0],0] plm:base:setup_vm only HNP in
>>>> allocation
>>>> [node08.cluster:26952] mca:rmaps: mapping job [56581,1]
>>>> [node08.cluster:26952] mca:rmaps: creating new map for job [56581,1]
>>>> [node08.cluster:26952] mca:rmaps:ppr: job [56581,1] not using ppr
> mapper
>>>> [node08.cluster:26952] [[56581,0],0] rmaps:seq mapping job [56581,1]
>>>> [node08.cluster:26952] mca:rmaps:seq: job [56581,1] not using seq
> mapper
>>>> [node08.cluster:26952] mca:rmaps:resilient: cannot perform initial map
> of
>>>> job [56581,1] - no fault groups
>>>> [node08.cluster:26952] mca:rmaps:mindist: job [56581,1] not using
> mindist
>>>> mapper
>>>> [node08.cluster:26952] mca:rmaps:rr: mapping job [56581,1]
>>>> [node08.cluster:26952] [[56581,0],0] Starting with 1 nodes in list
>>>> [node08.cluster:26952] [[56581,0],0] Filtering thru apps
>>>> [node08.cluster:26952] [[56581,0],0] Retained 1 nodes in list
>>>> [node08.cluster:26952] [[56581,0],0] Removing node node08 slots 0
> inuse 0
>>>>
>>>> From this result, I guess it's related to oversubscribe.
>>>> So I added "-oversubscribe" and rerun, then it worked well as show
> below:
>>>>
>>>> [node08.cluster:27019] [[56774,0],0] Starting with 1 nodes in list
>>>> [node08.cluster:27019] [[56774,0],0] Filtering thru apps
>>>> [node08.cluster:27019] [[56774,0],0] Retained 1 nodes in list
>>>> [node08.cluster:27019] AVAILABLE NODES FOR MAPPING:
>>>> [node08.cluster:27019] node: node08 daemon: 0
>>>> [node08.cluster:27019] [[56774,0],0] Starting bookmark at node node08
>>>> [node08.cluster:27019] [[56774,0],0] Starting at node node08
>>>> [node08.cluster:27019] mca:rmaps:rr: mapping by slot for job [56774,1]
>>>> slots 1 num_procs 8
>>>> [node08.cluster:27019] mca:rmaps:rr:slot working node node08
>>>> [node08.cluster:27019] mca:rmaps:rr:slot node node08 is full -
> skipping
>>>> [node08.cluster:27019] mca:rmaps:rr:slot job [56774,1] is
> oversubscribed -
>>>> performing second pass
>>>> [node08.cluster:27019] mca:rmaps:rr:slot working node node08
>>>> [node08.cluster:27019] mca:rmaps:rr:slot adding up to 8 procs to node
>>>> node08
>>>> [node08.cluster:27019] mca:rmaps:base: computing vpids by slot for job
>>>> [56774,1]
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 0 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 1 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 2 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 3 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 4 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 5 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 6 to node node08
>>>> [node08.cluster:27019] mca:rmaps:base: assigning rank 7 to node node08
>>>>
>>>> I think something is wrong with treatment of oversubscription, which
> might
>>>> be
>>>> related to "#3893: LAMA mapper has problems"
>>>>
>>>> tmishima
>>>>
>>>>> Hmmm...looks like we aren't getting your allocation. Can you rerun
> and
>>>> add -mca ras_base_verbose 50?
>>>>>
>>>>> On Nov 12, 2013, at 11:30 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Ralph,
>>>>>>
>>>>>> Here is the output of "-mca plm_base_verbose 5".
>>>>>>
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Querying component
> [rsh]
>>>>>> [node08.cluster:23573] [[INVALID],INVALID] plm:rsh_lookup on
>>>>>> agent /usr/bin/rsh path NULL
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Query of component
> [rsh]
>>>> set
>>>>>> priority to 10
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Querying component
>>>> [slurm]
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Skipping component
>>>> [slurm].
>>>>>> Query failed to return a module
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Querying component
> [tm]
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Query of component
> [tm]
>>>> set
>>>>>> priority to 75
>>>>>> [node08.cluster:23573] mca:base:select:( plm) Selected component
> [tm]
>>>>>> [node08.cluster:23573] plm:base:set_hnp_name: initial bias 23573
>>>> nodename
>>>>>> hash 85176670
>>>>>> [node08.cluster:23573] plm:base:set_hnp_name: final jobfam 59480
>>>>>> [node08.cluster:23573] [[59480,0],0] plm:base:receive start comm
>>>>>> [node08.cluster:23573] [[59480,0],0] plm:base:setup_job
>>>>>> [node08.cluster:23573] [[59480,0],0] plm:base:setup_vm
>>>>>> [node08.cluster:23573] [[59480,0],0] plm:base:setup_vm creating map
>>>>>> [node08.cluster:23573] [[59480,0],0] plm:base:setup_vm only HNP in
>>>>>> allocation
>>>>>>
>>>>
> --------------------------------------------------------------------------
>>>>>> All nodes which are allocated for this job are already filled.
>>>>>>
>>>>
> --------------------------------------------------------------------------
>>>>>>
>>>>>> Here, openmpi's configuration is as follows:
>>>>>>
>>>>>> ./configure \
>>>>>> --prefix=/home/mishima/opt/mpi/openmpi-1.7.4a1-pgi13.10 \
>>>>>> --with-tm \
>>>>>> --with-verbs \
>>>>>> --disable-ipv6 \
>>>>>> --disable-vt \
>>>>>> --enable-debug \
>>>>>> CC=pgcc CFLAGS="-tp k8-64e" \
>>>>>> CXX=pgCC CXXFLAGS="-tp k8-64e" \
>>>>>> F77=pgfortran FFLAGS="-tp k8-64e" \
>>>>>> FC=pgfortran FCFLAGS="-tp k8-64e"
>>>>>>
>>>>>>> Hi Ralph,
>>>>>>>
>>>>>>> Okey, I can help you. Please give me some time to report the
> output.
>>>>>>>
>>>>>>> Tetsuya Mishima
>>>>>>>
>>>>>>>> I can try, but I have no way of testing Torque any more - so all I
>>>> can
>>>>>> do
>>>>>>> is a code review. If you can build --enable-debug and add -mca
>>>>>>> plm_base_verbose 5 to your cmd line, I'd appreciate seeing the
>>>>>>>> output.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Nov 12, 2013, at 9:58 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Ralph,
>>>>>>>>>
>>>>>>>>> Thank you for your quick response.
>>>>>>>>>
>>>>>>>>> I'd like to report one more regressive issue about Torque support
> of
>>>>>>>>> openmpi-1.7.4a1r29646, which might be related to "#3893: LAMA
> mapper
>>>>>>>>> has problems" I reported a few days ago.
>>>>>>>>>
>>>>>>>>> The script below does not work with openmpi-1.7.4a1r29646,
>>>>>>>>> although it worked with openmpi-1.7.3 as I told you before.
>>>>>>>>>
>>>>>>>>> #!/bin/sh
>>>>>>>>> #PBS -l nodes=node08:ppn=8
>>>>>>>>> export OMP_NUM_THREADS=1
>>>>>>>>> cd $PBS_O_WORKDIR
>>>>>>>>> cp $PBS_NODEFILE pbs_hosts
>>>>>>>>> NPROCS=`wc -l < pbs_hosts`
>>>>>>>>> mpirun -machinefile pbs_hosts -np ${NPROCS} -report-bindings
>>>> -bind-to
>>>>>>> core
>>>>>>>>> Myprog
>>>>>>>>>
>>>>>>>>> If I drop "-machinefile pbs_hosts -np ${NPROCS} ", then it works
>>>>>> fine.
>>>>>>>>> Since this happens without lama request, I guess it's not the
>>>> problem
>>>>>>>>> in lama itself. Anyway, please look into this issue as well.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Tetsuya Mishima
>>>>>>>>>
>>>>>>>>>> Done - thanks!
>>>>>>>>>>
>>>>>>>>>> On Nov 12, 2013, at 7:35 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Dear openmpi developers,
>>>>>>>>>>>
>>>>>>>>>>> I got a segmentation fault in traial use of
> openmpi-1.7.4a1r29646
>>>>>>> built
>>>>>>>>> by
>>>>>>>>>>> PGI13.10 as shown below:
>>>>>>>>>>>
>>>>>>>>>>> [mishima@manage testbed-openmpi-1.7.3]$ mpirun -np 4
>>>> -cpus-per-proc
>>>>>> 2
>>>>>>>>>>> -report-bindings mPre
>>>>>>>>>>> [manage.cluster:23082] MCW rank 2 bound to socket 0[core 4[hwt
>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 5[hwt 0]]: [././././B/B][./././././.]
>>>>>>>>>>> [manage.cluster:23082] MCW rank 3 bound to socket 1[core 6[hwt
>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 7[hwt 0]]: [./././././.][B/B/./././.]
>>>>>>>>>>> [manage.cluster:23082] MCW rank 0 bound to socket 0[core 0[hwt
>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 1[hwt 0]]: [B/B/./././.][./././././.]
>>>>>>>>>>> [manage.cluster:23082] MCW rank 1 bound to socket 0[core 2[hwt
>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 3[hwt 0]]: [././B/B/./.][./././././.]
>>>>>>>>>>> [manage:23082] *** Process received signal ***
>>>>>>>>>>> [manage:23082] Signal: Segmentation fault (11)
>>>>>>>>>>> [manage:23082] Signal code: Address not mapped (1)
>>>>>>>>>>> [manage:23082] Failing at address: 0x34
>>>>>>>>>>> [manage:23082] *** End of error message ***
>>>>>>>>>>> Segmentation fault (core dumped)
>>>>>>>>>>>
>>>>>>>>>>> [mishima@manage testbed-openmpi-1.7.3]$ gdb mpirun core.23082
>>>>>>>>>>> GNU gdb (GDB) CentOS (7.0.1-42.el5.centos.1)
>>>>>>>>>>> Copyright (C) 2009 Free Software Foundation, Inc.
>>>>>>>>>>> ...
>>>>>>>>>>> Core was generated by `mpirun -np 4 -cpus-per-proc 2
>>>>>> -report-bindings
>>>>>>>>>>> mPre'.
>>>>>>>>>>> Program terminated with signal 11, Segmentation fault.
>>>>>>>>>>> #0 0x00002b5f861c9c4f in recv_connect (mod=0x5f861ca20b00007f,
>>>>>>>>> sd=32767,
>>>>>>>>>>> hdr=0x1ca20b00007fff25) at ./oob_tcp.c:631
>>>>>>>>>>> 631 peer = OBJ_NEW(mca_oob_tcp_peer_t);
>>>>>>>>>>> (gdb) where
>>>>>>>>>>> #0 0x00002b5f861c9c4f in recv_connect (mod=0x5f861ca20b00007f,
>>>>>>>>> sd=32767,
>>>>>>>>>>> hdr=0x1ca20b00007fff25) at ./oob_tcp.c:631
>>>>>>>>>>> #1 0x00002b5f861ca20b in recv_handler (sd=1778385023,
>>>> flags=32767,
>>>>>>>>>>> cbdata=0x8eb06a00007fff25) at ./oob_tcp.c:760
>>>>>>>>>>> #2 0x00002b5f848eb06a in event_process_active_single_queue
>>>>>>>>>>> (base=0x5f848eb27000007f, activeq=0x848eb27000007fff)
>>>>>>>>>>> at ./event.c:1366
>>>>>>>>>>> #3 0x00002b5f848eb270 in event_process_active
>>>>>>>>> (base=0x5f848eb84900007f)
>>>>>>>>>>> at ./event.c:1435
>>>>>>>>>>> #4 0x00002b5f848eb849 in opal_libevent2021_event_base_loop
>>>>>>>>>>> (base=0x4077a000007f, flags=32767) at ./event.c:1645
>>>>>>>>>>> #5 0x00000000004077a0 in orterun (argc=7, argv=0x7fff25bbd4a8)
>>>>>>>>>>> at ./orterun.c:1030
>>>>>>>>>>> #6 0x00000000004067fb in main (argc=7, argv=0x7fff25bbd4a8)
>>>>>>>>> at ./main.c:13
>>>>>>>>>>> (gdb) quit
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The line 627 in orte/mca/oob/tcp/oob_tcp.c is apparently
>>>>>> unnecessary,
>>>>>>>>> which
>>>>>>>>>>> causes the segfault.
>>>>>>>>>>>
>>>>>>>>>>> 624 /* lookup the corresponding process */
>>>>>>>>>>> 625 peer = mca_oob_tcp_peer_lookup(mod, &hdr->origin);
>>>>>>>>>>> 626 if (NULL == peer) {
>>>>>>>>>>> 627 ui64 = (uint64_t*)(&peer->name);
>>>>>>>>>>> 628 opal_output_verbose(OOB_TCP_DEBUG_CONNECT,
>>>>>>>>>>> orte_oob_base_framework.framework_output,
>>>>>>>>>>> 629 "%s mca_oob_tcp_recv_connect:
>>>>>>>>>>> connection from new peer",
>>>>>>>>>>> 630 ORTE_NAME_PRINT
>>>>>>> (ORTE_PROC_MY_NAME));
>>>>>>>>>>> 631 peer = OBJ_NEW(mca_oob_tcp_peer_t);
>>>>>>>>>>> 632 peer->mod = mod;
>>>>>>>>>>> 633 peer->name = hdr->origin;
>>>>>>>>>>> 634 peer->state = MCA_OOB_TCP_ACCEPTING;
>>>>>>>>>>> 635 ui64 = (uint64_t*)(&peer->name);
>>>>>>>>>>> 636 if (OPAL_SUCCESS !=
> opal_hash_table_set_value_uint64
>>>>>>>>> (&mod->
>>>>>>>>>>> peers, (*ui64), peer)) {
>>>>>>>>>>> 637 OBJ_RELEASE(peer);
>>>>>>>>>>> 638 return;
>>>>>>>>>>> 639 }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please fix this mistake in the next release.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
