Actually, it looks like it would happen with hetero-nodes set - only required
that at least two nodes have the same architecture. So you might want to give
the trunk a shot as it may well now be fixed.
On Dec 19, 2013, at 8:35 AM, Ralph Castain <r...@open-mpi.org> wrote:
> Hmmm...not having any luck tracking this down yet. If anything, based on what
> I saw in the code, I would have expected it to fail when hetero-nodes was
> false, not the other way around.
>
> I'll keep poking around - just wanted to provide an update.
>
> On Dec 19, 2013, at 12:54 AM, tmish...@jcity.maeda.co.jp wrote:
>
>>
>>
>> Hi Ralph, sorry for intersecting post.
>>
>> Your advice about -hetero-nodes in other thread gives me a hint.
>>
>> I already put "orte_hetero_nodes = 1" in my mca-params.conf, because
>> you told me a month ago that my environment would need this option.
>>
>> Removing this line from mca-params.conf, then it works.
>> In other word, you can replicate it by adding -hetero-nodes as
>> shown below.
>>
>> qsub: job 8364.manage.cluster completed
>> [mishima@manage mpi]$ qsub -I -l nodes=2:ppn=8
>> qsub: waiting for job 8365.manage.cluster to start
>> qsub: job 8365.manage.cluster ready
>>
>> [mishima@node11 ~]$ ompi_info --all | grep orte_hetero_nodes
>> MCA orte: parameter "orte_hetero_nodes" (current value:
>> "false", data source: default, level: 9 dev/all,
>> type: bool)
>> [mishima@node11 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
>> myprog
>> [node11.cluster:27895] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>> [node11.cluster:27895] MCW rank 1 bound to socket 1[core 4[hwt 0]], socket
>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>> [node12.cluster:24891] MCW rank 3 bound to socket 1[core 4[hwt 0]], socket
>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>> [node12.cluster:24891] MCW rank 2 bound to socket 0[core 0[hwt 0]], socket
>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>> Hello world from process 0 of 4
>> Hello world from process 1 of 4
>> Hello world from process 2 of 4
>> Hello world from process 3 of 4
>> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
>> -hetero-nodes myprog
>> --------------------------------------------------------------------------
>> A request was made to bind to that would result in binding more
>> processes than cpus on a resource:
>>
>> Bind to: CORE
>> Node: node12
>> #processes: 2
>> #cpus: 1
>>
>> You can override this protection by adding the "overload-allowed"
>> option to your binding directive.
>> --------------------------------------------------------------------------
>>
>>
>> As far as I checked, data->num_bound seems to become bad in bind_downwards,
>> when I put "-hetero-nodes". I hope you can clear the problem.
>>
>> Regards,
>> Tetsuya Mishima
>>
>>
>>> Yes, it's very strange. But I don't think there's any chance that
>>> I have < 8 actual cores on the node. I guess that you cat replicate
>>> it with SLURM, please try it again.
>>>
>>> I changed to use node10 and node11, then I got the warning against
>>> node11.
>>>
>>> Furthermore, just as an information for you, I tried to add
>>> "-bind-to core:overload-allowed", then it worked as shown below.
>>> But I think node11 is never overloaded because it has 8 cores.
>>>
>>> qsub: job 8342.manage.cluster completed
>>> [mishima@manage ~]$ qsub -I -l nodes=node10:ppn=8+node11:ppn=8
>>> qsub: waiting for job 8343.manage.cluster to start
>>> qsub: job 8343.manage.cluster ready
>>>
>>> [mishima@node10 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>> [mishima@node10 demos]$ cat $PBS_NODEFILE
>>> node10
>>> node10
>>> node10
>>> node10
>>> node10
>>> node10
>>> node10
>>> node10
>>> node11
>>> node11
>>> node11
>>> node11
>>> node11
>>> node11
>>> node11
>>> node11
>>> [mishima@node10 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
>>> myprog
>>>
>> --------------------------------------------------------------------------
>>> A request was made to bind to that would result in binding more
>>> processes than cpus on a resource:
>>>
>>> Bind to: CORE
>>> Node: node11
>>> #processes: 2
>>> #cpus: 1
>>>
>>> You can override this protection by adding the "overload-allowed"
>>> option to your binding directive.
>>>
>> --------------------------------------------------------------------------
>>> [mishima@node10 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
>>> -bind-to core:overload-allowed myprog
>>> [node10.cluster:27020] MCW rank 0 bound to socket 0[core 0[hwt 0]],
>> socket
>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>>> [node10.cluster:27020] MCW rank 1 bound to socket 1[core 4[hwt 0]],
>> socket
>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>>> [node11.cluster:26597] MCW rank 3 bound to socket 1[core 4[hwt 0]],
>> socket
>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>>> [node11.cluster:26597] MCW rank 2 bound to socket 0[core 0[hwt 0]],
>> socket
>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>>> Hello world from process 1 of 4
>>> Hello world from process 0 of 4
>>> Hello world from process 3 of 4
>>> Hello world from process 2 of 4
>>>
>>> Regards,
>>> Tetsuya Mishima
>>>
>>>
>>>> Very strange - I can't seem to replicate it. Is there any chance that
>> you
>>> have < 8 actual cores on node12?
>>>>
>>>>
>>>> On Dec 18, 2013, at 4:53 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>
>>>>>
>>>>>
>>>>> Hi Ralph, sorry for confusing you.
>>>>>
>>>>> At that time, I cut and paste the part of "cat $PBS_NODEFILE".
>>>>> I guess I didn't paste the last line by my mistake.
>>>>>
>>>>> I retried the test and below one is exactly what I got when I did the
>>> test.
>>>>>
>>>>> [mishima@manage ~]$ qsub -I -l nodes=node11:ppn=8+node12:ppn=8
>>>>> qsub: waiting for job 8338.manage.cluster to start
>>>>> qsub: job 8338.manage.cluster ready
>>>>>
>>>>> [mishima@node11 ~]$ cat $PBS_NODEFILE
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node11
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> node12
>>>>> [mishima@node11 ~]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
>>> myprog
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>> A request was made to bind to that would result in binding more
>>>>> processes than cpus on a resource:
>>>>>
>>>>> Bind to: CORE
>>>>> Node: node12
>>>>> #processes: 2
>>>>> #cpus: 1
>>>>>
>>>>> You can override this protection by adding the "overload-allowed"
>>>>> option to your binding directive.
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>
>>>>> Regards,
>>>>>
>>>>> Tetsuya Mishima
>>>>>
>>>>>> I removed the debug in #2 - thanks for reporting it
>>>>>>
>>>>>> For #1, it actually looks to me like this is correct. If you look at
>>> your
>>>>> allocation, there are only 7 slots being allocated on node12, yet you
>>> have
>>>>> asked for 8 cpus to be assigned (2 procs with 2
>>>>>> cpus/proc). So the warning is in fact correct
>>>>>>
>>>>>>
>>>>>> On Dec 18, 2013, at 4:04 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Ralph, I found that openmpi-1.7.4rc1 was already uploaded. So
>> I'd
>>>>> like
>>>>>>> to report
>>>>>>> 3 issues mainly regarding -cpus-per-proc.
>>>>>>>
>>>>>>> 1) When I use 2 nodes(node11,node12), which has 8 cores each(= 2
>>>>> sockets X
>>>>>>> 4 cores/socket),
>>>>>>> it starts to produce the error again as shown below. At least,
>>>>>>> openmpi-1.7.4a1r29646 did
>>>>>>> work well.
>>>>>>>
>>>>>>> [mishima@manage ~]$ qsub -I -l nodes=2:ppn=8
>>>>>>> qsub: waiting for job 8336.manage.cluster to start
>>>>>>> qsub: job 8336.manage.cluster ready
>>>>>>>
>>>>>>> [mishima@node11 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>> [mishima@node11 demos]$ cat $PBS_NODEFILE
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node11
>>>>>>> node12
>>>>>>> node12
>>>>>>> node12
>>>>>>> node12
>>>>>>> node12
>>>>>>> node12
>>>>>>> node12
>>>>>>> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4
>>> -report-bindings
>>>>>>> myprog
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>> A request was made to bind to that would result in binding more
>>>>>>> processes than cpus on a resource:
>>>>>>>
>>>>>>> Bind to: CORE
>>>>>>> Node: node12
>>>>>>> #processes: 2
>>>>>>> #cpus: 1
>>>>>>>
>>>>>>> You can override this protection by adding the "overload-allowed"
>>>>>>> option to your binding directive.
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>
>>>>>>> Of course it works well using only one node.
>>>>>>>
>>>>>>> [mishima@node11 demos]$ mpirun -np 2 -cpus-per-proc 4
>>> -report-bindings
>>>>>>> myprog
>>>>>>> [node11.cluster:26238] MCW rank 0 bound to socket 0[core 0[hwt 0]],
>>>>> socket
>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>>>>>>> [node11.cluster:26238] MCW rank 1 bound to socket 1[core 4[hwt 0]],
>>>>> socket
>>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>>>>>>> Hello world from process 1 of 2
>>>>>>> Hello world from process 0 of 2
>>>>>>>
>>>>>>>
>>>>>>> 2) Adding "-bind-to numa", it works but the message "bind:upward
>>> target
>>>>>>> NUMANode type NUMANode" appears.
>>>>>>> As far as I remember, I didn't see such a kind of message before.
>>>>>>>
>>>>>>> mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4
>> -report-bindings
>>>>>>> -bind-to numa myprog
>>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
>> type
>>>>>>> NUMANode
>>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
>> type
>>>>>>> NUMANode
>>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
>> type
>>>>>>> NUMANode
>>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
>> type
>>>>>>> NUMANode
>>>>>>> [node11.cluster:26260] MCW rank 0 bound to socket 0[core 0[hwt 0]],
>>>>> socket
>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>>>>>>> [node11.cluster:26260] MCW rank 1 bound to socket 1[core 4[hwt 0]],
>>>>> socket
>>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>>>>>>> [node12.cluster:23607] MCW rank 3 bound to socket 1[core 4[hwt 0]],
>>>>> socket
>>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
>>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
>>>>>>> [node12.cluster:23607] MCW rank 2 bound to socket 0[core 0[hwt 0]],
>>>>> socket
>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
>>>>>>> Hello world from process 1 of 4
>>>>>>> Hello world from process 0 of 4
>>>>>>> Hello world from process 3 of 4
>>>>>>> Hello world from process 2 of 4
>>>>>>>
>>>>>>>
>>>>>>> 3) I use PGI compiler. It can not accept compiler switch
>>>>>>> "-Wno-variadic-macros", which is
>>>>>>> included in configure script.
>>>>>>>
>>>>>>> btl_usnic_CFLAGS="-Wno-variadic-macros"
>>>>>>>
>>>>>>> I removed this switch, then I could continue to build 1.7.4rc1.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Tetsuya Mishima
>>>>>>>
>>>>>>>
>>>>>>>> Hmmm...okay, I understand the scenario. Must be something in the
>>> algo
>>>>>>> when it only has one node, so it shouldn't be too hard to track
>> down.
>>>>>>>>
>>>>>>>> I'm off on travel for a few days, but will return to this when I
>> get
>>>>>>> back.
>>>>>>>>
>>>>>>>> Sorry for delay - will try to look at this while I'm gone, but
>> can't
>>>>>>> promise anything :-(
>>>>>>>>
>>>>>>>>
>>>>>>>> On Dec 10, 2013, at 6:58 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Ralph, sorry for confusing.
>>>>>>>>>
>>>>>>>>> We usually logon to "manage", which is our control node.
>>>>>>>>> From manage, we submit job or enter a remote node such as
>>>>>>>>> node03 by torque interactive mode(qsub -I).
>>>>>>>>>
>>>>>>>>> At that time, instead of torque, I just did rsh to node03 from
>>> manage
>>>>>>>>> and ran myprog on the node. I hope you could understand what I
>> did.
>>>>>>>>>
>>>>>>>>> Now, I retried with "-host node03", which still causes the
>> problem:
>>>>>>>>> (I comfirmed local run on manage caused the same problem too)
>>>>>>>>>
>>>>>>>>> [mishima@manage ~]$ rsh node03
>>>>>>>>> Last login: Wed Dec 11 11:38:57 from manage
>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>>>> [mishima@node03 demos]$
>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -host node03
>> -report-bindings
>>>>>>>>> -cpus-per-proc 4 -map-by socket myprog
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>> A request was made to bind to that would result in binding more
>>>>>>>>> processes than cpus on a resource:
>>>>>>>>>
>>>>>>>>> Bind to: CORE
>>>>>>>>> Node: node03
>>>>>>>>> #processes: 2
>>>>>>>>> #cpus: 1
>>>>>>>>>
>>>>>>>>> You can override this protection by adding the "overload-allowed"
>>>>>>>>> option to your binding directive.
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>>
>>>>>>>>> It' strange, but I have to report that "-map-by socket:span"
>> worked
>>>>>>> well.
>>>>>>>>>
>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -host node03
>> -report-bindings
>>>>>>>>> -cpus-per-proc 4 -map-by socket:span myprog
>>>>>>>>> [node03.cluster:11871] MCW rank 2 bound to socket 1[core 8[hwt
>> 0]],
>>>>>>> socket
>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
>>>>>>>>> ocket 1[core 11[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 3 bound to socket 1[core 12[hwt
>>> 0]],
>>>>>>> socket
>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
>>>>>>>>> socket 1[core 15[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 4 bound to socket 2[core 16[hwt
>>> 0]],
>>>>>>> socket
>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
>>>>>>>>> socket 2[core 19[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 5 bound to socket 2[core 20[hwt
>>> 0]],
>>>>>>> socket
>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
>>>>>>>>> socket 2[core 23[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 6 bound to socket 3[core 24[hwt
>>> 0]],
>>>>>>> socket
>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
>>>>>>>>> socket 3[core 27[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 7 bound to socket 3[core 28[hwt
>>> 0]],
>>>>>>> socket
>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
>>>>>>>>> socket 3[core 31[hwt 0]]:
>>>>>>>>>
>>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
>>>>>>>>> [node03.cluster:11871] MCW rank 0 bound to socket 0[core 0[hwt
>> 0]],
>>>>>>> socket
>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>
>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>> [node03.cluster:11871] MCW rank 1 bound to socket 0[core 4[hwt
>> 0]],
>>>>>>> socket
>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>
>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Tetsuya Mishima
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Dec 10, 2013, at 6:05 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>
>>>>>>>>>>> I tried again with -cpus-per-proc 2 as shown below.
>>>>>>>>>>> Here, I found that "-map-by socket:span" worked well.
>>>>>>>>>>>
>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
>>>>> -cpus-per-proc
>>>>>>> 2
>>>>>>>>>>> -map-by socket:span myprog
>>>>>>>>>>> [node03.cluster:10879] MCW rank 2 bound to socket 1[core 8[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 9[hwt 0]]: [./././././././.][B/B/././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 3 bound to socket 1[core 10[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 11[hwt 0]]: [./././././././.][././B/B
>>>>>>>>>>> /./././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 4 bound to socket 2[core 16[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 2[core 17[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /./././.][B/B/./././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 5 bound to socket 2[core 18[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 2[core 19[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /./././.][././B/B/./././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 6 bound to socket 3[core 24[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 3[core 25[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /./././.][./././././././.][B/B/./././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 7 bound to socket 3[core 26[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 3[core 27[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /./././.][./././././././.][././B/B/./././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 0 bound to socket 0[core 0[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 1[hwt 0]]: [B/B/./././././.][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10879] MCW rank 1 bound to socket 0[core 2[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 3[hwt 0]]: [././B/B/./././.][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>> Hello world from process 5 of 8> >>>>>>> Hello world from
>> process 3 of 8
>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
>>>>> -cpus-per-proc
>>>>>>> 2
>>>>>>>>>>> -map-by socket myprog
>>>>>>>>>>> [node03.cluster:10921] MCW rank 2 bound to socket 0[core 4[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 5[hwt 0]]: [././././B/B/./.][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 3 bound to socket 0[core 6[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 7[hwt 0]]: [././././././B/B][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 4 bound to socket 1[core 8[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 9[hwt 0]]: [./././././././.][B/B/././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 5 bound to socket 1[core 10[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 11[hwt 0]]: [./././././././.][././B/B
>>>>>>>>>>> /./././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 6 bound to socket 1[core 12[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 13[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /B/B/./.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 7 bound to socket 1[core 14[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 15[hwt 0]]: [./././././././.][./././.
>>>>>>>>>>> /././B/B][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 0 bound to socket 0[core 0[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 1[hwt 0]]: [B/B/./././././.][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:10921] MCW rank 1 bound to socket 0[core 2[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 3[hwt 0]]: [././B/B/./././.][././././.
>>>>>>>>>>> /././.][./././././././.][./././././././.]
>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>>>>
>>>>>>>>>>> "-np 8" and "-cpus-per-proc 4" just filled all sockets.
>>>>>>>>>>> In this case, I guess "-map-by socket:span" and "-map-by
>> socket"
>>>>> has
>>>>>>>>> same
>>>>>>>>>>> meaning.
>>>>>>>>>>> Therefore, there's no problem about that. Sorry for distubing.
>>>>>>>>>>
>>>>>>>>>> No problem - glad you could clear that up :-)
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> By the way, through this test, I found another problem.
>>>>>>>>>>> Without torque manager and just using rsh, it causes the same
>>> error
>>>>>>>>> like
>>>>>>>>>>> below:
>>>>>>>>>>>
>>>>>>>>>>> [mishima@manage openmpi-1.7]$ rsh node03
>>>>>>>>>>> Last login: Wed Dec 11 09:42:02 from manage
>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
>>>>> -cpus-per-proc
>>>>>>> 4
>>>>>>>>>>> -map-by socket myprog
>>>>>>>>>>
>>>>>>>>>> I don't understand the difference here - you are simply starting
>>> it
>>>>>>> from>>>>> a different node? It looks like everything is expected to
>>> run local
>>>>> to
>>>>>>>>> mpirun, yes? So there is no rsh actually involved here.
>>>>>>>>>> Are you still running in an allocation?
>>>>>>>>>>
>>>>>>>>>> If you run this with "-host node03" on the cmd line, do you see
>>> the
>>>>>>> same
>>>>>>>>> problem?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>>>> A request was made to bind to that would result in binding more
>>>>>>>>>>> processes than cpus on a resource:
>>>>>>>>>>>
>>>>>>>>>>> Bind to: CORE
>>>>>>>>>>> Node: node03
>>>>>>>>>>> #processes: 2
>>>>>>>>>>> #cpus: 1
>>>>>>>>>>>
>>>>>>>>>>> You can override this protection by adding the
>> "overload-allowed"
>>>>>>>>>>> option to your binding directive.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>>>> [mishima@node03 demos]$
>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
>>>>> -cpus-per-proc
>>>>>>> 4
>>>>>>>>>>> myprog
>>>>>>>>>>> [node03.cluster:11036] MCW rank 2 bound to socket 1[core 8[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
>>>>>>>>>>>
>>>>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 3 bound to socket 1[core 12[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
>>>>>>>>>>> socket 1[core 15[hwt 0]]:
>>>>>>>>>>>
>>>>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 4 bound to socket 2[core 16[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
>>>>>>>>>>> socket 2[core 19[hwt 0]]:
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 5 bound to socket 2[core 20[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
>>>>>>>>>>> socket 2[core 23[hwt 0]]:
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 6 bound to socket 3[core 24[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
>>>>>>>>>>> socket 3[core 27[hwt 0]]:>>>>>
>>>>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 7 bound to socket 3[core 28[hwt
>>>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
>>>>>>>>>>> socket 3[core 31[hwt 0]]:
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 0 bound to socket 0[core 0[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>
>>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>> [node03.cluster:11036] MCW rank 1 bound to socket 0[core 4[hwt
>>> 0]],
>>>>>>>>> socket
>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>
>>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>
>>>>>>>>>>>> Hmmm...that's strange. I only have 2 sockets on my system, but
>>> let
>>>>>>> me
>>>>>>>>>>> poke around a bit and see what might be happening.
>>>>>>>>>>>>
>>>>>>>>>>>> On Dec 10, 2013, at 4:47 PM, tmish...@jcity.maeda.co.jp wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks. I didn't know the meaning of "socket:span".
>>>>>>>>>>>>>
>>>>>>>>>>>>> But it still causes the problem, which seems socket:span
>>> doesn't
>>>>>>>>> work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [mishima@manage demos]$ qsub -I -l nodes=node03:ppn=32
>>>>>>>>>>>>> qsub: waiting for job 8265.manage.cluster to start
>>>>>>>>>>>>> qsub: job 8265.manage.cluster ready
>>>>>>>>>>>>>
>>>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
>>>>>>> -cpus-per-proc
>>>>>>>>> 4
>>>>>>>>>>>>> -map-by socket:span myprog
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 2 bound to socket 1[core 8
>> [hwt
>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
>>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 3 bound to socket 1[core 12
>> [hwt
>>>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
>>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 4 bound to socket 2[core 16
>> [hwt
>>>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
>>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 5 bound to socket 2[core 20
>> [hwt
>>>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
>>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 6 bound to socket 3[core 24
>> [hwt
>>>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
>>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 7 bound to socket 3[core 28
>> [hwt
>>>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
>>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 0 bound to socket 0[core 0
>> [hwt
>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>> [node03.cluster:10262] MCW rank 1 bound to socket 0[core 4
>> [hwt
>>>>> 0]],
>>>>>>>>>>> socket
>>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>>
>>>>>>>
>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>>
>>>>>>>>>>>>>> No, that is actually correct. We map a socket until full,
>> then
>>>>>>> move
>>>>>>>>> to
>>>>>>>>>>>>> the next. What you want is --map-by socket:span
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Dec 10, 2013, at 3:42 PM, tmish...@jcity.maeda.co.jp
>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I had a time to try your patch yesterday using
>>>>>>>>> openmpi-1.7.4a1r29646.
>>>>>>>>>>>>>>>>>>>>>> It stopped the error but unfortunately "mapping by
>>>>>>> socket" itself
>>>>>>>>>>>>> didn't
>>>>>>>>>>>>>>> work
>>>>>>>>>>>>>>> well as shown bellow:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [mishima@manage demos]$ qsub -I -l nodes=1:ppn=32
>>>>>>>>>>>>>>> qsub: waiting for job 8260.manage.cluster to start
>>>>>>>>>>>>>>> qsub: job 8260.manage.cluster ready
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [mishima@node04 ~]$ cd ~/Desktop/openmpi-1.7/demos/
>>>>>>>>>>>>>>> [mishima@node04 demos]$ mpirun -np 8 -report-bindings
>>>>>>>>> -cpus-per-proc
>>>>>>>>>>> 4
>>>>>>>>>>>>>>> -map-by socket myprog
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 2 bound to socket 1[core 8
>>> [hwt
>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
>>>>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 3 bound to socket 1[core 12
>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
>>>>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 4 bound to socket 2[core 16
>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
>>>>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 5 bound to socket 2[core 20
>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
>>>>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 6 bound to socket 3[core 24
>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
>>>>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 7 bound to socket 3[core 28
>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
>>>>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 0 bound to socket 0[core 0
>>> [hwt
>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 1 bound to socket 0[core 4
>>> [hwt
>>>>>>> 0]],
>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>> Hello world from process 2 of 8
>>>>>>>>>>>>>>> Hello world from process 1 of 8
>>>>>>>>>>>>>>> Hello world from process 3 of 8
>>>>>>>>>>>>>>> Hello world from process 0 of 8
>>>>>>>>>>>>>>> Hello world from process 6 of 8
>>>>>>>>>>>>>>> Hello world from process 5 of 8
>>>>>>>>>>>>>>> Hello world from process 4 of 8
>>>>>>>>>>>>>>> Hello world from process 7 of 8
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think this should be like this:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rank 00
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>> rank 01
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>> rank 02
>>>>>>>>>>>>>>>
>>>>>>>>>
>>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I fixed this under the trunk (was an issue regardless of
>> RM)
>>>>> and
>>>>>>>>>>> have
>>>>>>>>>>>>>>> scheduled it for 1.7.4.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Nov 25, 2013, at 4:22 PM, tmish...@jcity.maeda.co.jp
>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you very much for your quick response.>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm afraid to say that I found one more issuse...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's not so serious. Please check it when you have a lot
>> of
>>>>>>> time.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The problem is cpus-per-proc with -map-by option under
>>> Torque
>>>>>>>>>>>>> manager.
>>>>>>>>>>>>>>>>> It doesn't work as shown below. I guess you can get the
>>> same
>>>>>>>>>>>>>>>>> behaviour under Slurm manager.
>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Of course, if I remove -map-by option, it works quite
>> well.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [mishima@manage testbed2]$ qsub -I -l nodes=1:ppn=32
>>>>>>>>>>>>>>>>> qsub: waiting for job 8116.manage.cluster to start
>>>>>>>>>>>>>>>>> qsub: job 8116.manage.cluster ready
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Ducom/testbed2
>>>>>>>>>>>>>>>>> [mishima@node03 testbed2]$ mpirun -np 8 -report-bindings
>>>>>>>>>>>>> -cpus-per-proc
>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>>>> -map-by socket mPre
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>>> A request was made to bind to that would result in
>> binding
>>>>> more
>>>>>>>>>>>>>>>>> processes than cpus on a resource:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Bind to: CORE
>>>>>>>>>>>>>>>>> Node: node03>>>>>>> #processes: 2
>>>>>>>>>>>>>>>>> #cpus: 1
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> You can override this protection by adding the
>>>>>>> "overload-allowed"
>>>>>>>>>>>>>>>>> option to your binding directive.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> --------------------------------------------------------------------------
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [mishima@node03 testbed2]$ mpirun -np 8 -report-bindings
>>>>>>>>>>>>> -cpus-per-proc
>>>>>>>>>>>>>>> 4
>>>>>>>>>>>>>>>>> mPre
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 2 bound to socket 1[core
>> 8
>>>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
>>>>>>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 3 bound to socket 1[core
>> 12
>>>>> [hwt
>>>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
>>>>>>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 4 bound to socket 2[core
>> 16
>>>>> [hwt
>>>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
>>>>>>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 5 bound to socket 2[core
>> 20
>>>>> [hwt
>>>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
>>>>>>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 6 bound to socket 3[core
>> 24
>>>>> [hwt
>>>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
>>>>>>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 7 bound to socket 3[core
>> 28
>>>>> [hwt
>>>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
>>>>>>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>>>
>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]>>>>>>>>>>>>>
>>
>>> [node03.cluster:18128] MCW rank 0 bound to socket 0[core 0
>>>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
>>>>>>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 1 bound to socket 0[core
>> 4
>>>>> [hwt
>>>>>>>>> 0]],
>>>>>>>>>>>>>>> socket
>>>>>>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
>>>>>>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>> Regards,
>>>>>>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Fixed and scheduled to move to 1.7.4. Thanks again!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Nov 17, 2013, at 6:11 PM, Ralph Castain
>>>>> <r...@open-mpi.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks! That's precisely where I was going to look when
>> I
>>>>> had
>>>>>>>>>>>>> time :-)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'll update tomorrow.
>>>>>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Sun, Nov 17, 2013 at 7:01 PM,
>>>>>>>>>>> <tmish...@jcity.maeda.co.jp>wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Ralph,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is the continuous story of "Segmentation fault in
>>>>>>> oob_tcp.c
>>>>>>>>>>> of
>>>>>>>>>>>>>>>>>> openmpi-1.7.4a1r29646".
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I found the cause.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Firstly, I noticed that your hostfile can work and mine
>>> can
>>>>>>> not.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Your host file:
>>>>>>>>>>>>>>>>>> cat hosts
>>>>>>>>>>>>>>>>>> bend001 slots=12
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> My host file:
>>>>>>>>>>>>>>>>>> cat hosts
>>>>>>>>>>>>>>>>>> node08
>>>>>>>>>>>>>>>>>> node08
>>>>>>>>>>>>>>>>>> ...(total 8 lines)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I modified my script file to add "slots=1" to each line
>> of
>>>>> my
>>>>>>>>>>>>> hostfile
>>>>>>>>>>>>>>>>>> just before launching mpirun. Then it worked.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> My host file(modified):
>>>>>>>>>>>>>>>>>> cat hosts
>>>>>>>>>>>>>>>>>> node08 slots=1
>>>>>>>>>>>>>>>>>> node08 slots=1
>>>>>>>>>>>>>>>>>> ...(total 8 lines)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Secondary, I confirmed that there's a slight difference
>>>>>>> between
>>>>>>>>>>>>>>>>>> orte/util/hostfile/hostfile.c of 1.7.3 and that of
>>>>>>>>> 1.7.4a1r29646.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> $ diff
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>
>> hostfile.c.org ../../../../openmpi-1.7.3/orte/util/hostfile/hostfile.c
>>>>>>>>>>>>>>>>>> 394,401c394,399
>>>>>>>>>>>>>>>>>> < if (got_count) {
>>>>>>>>>>>>>>>>>> < node->slots_given = true;
>>>>>>>>>>>>>>>>>> < } else if (got_max) {
>>>>>>>>>>>>>>>>>> < node->slots = node->slots_max;
>>>>>>>>>>>>>>>>>> < node->slots_given = true;
>>>>>>>>>>>>>>>>>> < } else {
>>>>>>>>>>>>>>>>>> < /* should be set by obj_new, but just to be
>>> clear
>>>>> */
>>>>>>>>>>>>>>>>>> < node->slots_given = false;
>>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>>>> if (!got_count) {
>>>>>>>>>>>>>>>>>>> if (got_max) {
>>>>>>>>>>>>>>>>>>> node->slots = node->slots_max;
>>>>>>>>>>>>>>>>>>> } else {
>>>>>>>>>>>>>>>>>>> ++node->slots;>>>>>>>>>>>>> }
>>>>>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Finally, I added the line 402 below just as a tentative
>>>>> trial.
>>>>>>>>>>>>>>>>>> Then, it worked.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> cat -n orte/util/hostfile/hostfile.c:
>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>> 394 if (got_count) {
>>>>>>>>>>>>>>>>>> 395 node->slots_given = true;
>>>>>>>>>>>>>>>>>> 396 } else if (got_max) {
>>>>>>>>>>>>>>>>>> 397 node->slots = node->slots_max;
>>>>>>>>>>>>>>>>>> 398 node->slots_given = true;
>>>>>>>>>>>>>>>>>> 399 } else {
>>>>>>>>>>>>>>>>>> 400 /* should be set by obj_new, but just to be
>>>>> clear
>>>>>>>>> */
>>>>>>>>>>>>>>>>>> 401 node->slots_given
>>>>> = false;
>>>>>>>>>>>>>>>>>> 402 ++node->slots; /* added by tmishima */
>>>>>>>>>>>>>>>>>> 403 }
>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Please fix the problem properly, because it's just based
>>> on
>>>>> my
>>>>>>>>>>>>>>>>>> random guess. It's related to the treatment of hostfile
>>>>> where
>>>>>>>>>>> slots
>>>>>>>>>>>>>>>>>> information is not given.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>> Tetsuya Mishima
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>
>>>
>> http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________
>>
>>>
>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>> users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> users mailing list
>>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> users mailing list
>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
