Hi Ralph,
Thank you for your fix. It works for me.
Tetsuya Mishima
> Actually, it looks like it would happen with hetero-nodes set - only
required that at least two nodes have the same architecture. So you might
want to give the trunk a shot as it may well now be
> fixed.
>
>
> On Dec 19, 2013, at 8:35 AM, Ralph Castain <r...@open-mpi.org> wrote:
>
> > Hmmm...not having any luck tracking this down yet. If anything, based
on what I saw in the code, I would have expected it to fail when
hetero-nodes was false, not the other way around.
> >
> > I'll keep poking around - just wanted to provide an update.
> >
> > On Dec 19, 2013, at 12:54 AM, tmish...@jcity.maeda.co.jp wrote:
> >
> >>
> >>
> >> Hi Ralph, sorry for intersecting post.
> >>
> >> Your advice about -hetero-nodes in other thread gives me a hint.
> >>
> >> I already put "orte_hetero_nodes = 1" in my mca-params.conf, because
> >> you told me a month ago that my environment would need this option.
> >>
> >> Removing this line from mca-params.conf, then it works.
> >> In other word, you can replicate it by adding -hetero-nodes as
> >> shown below.
> >>
> >> qsub: job 8364.manage.cluster completed
> >> [mishima@manage mpi]$ qsub -I -l nodes=2:ppn=8
> >> qsub: waiting for job 8365.manage.cluster to start
> >> qsub: job 8365.manage.cluster ready
> >>
> >> [mishima@node11 ~]$ ompi_info --all | grep orte_hetero_nodes
> >> MCA orte: parameter "orte_hetero_nodes" (current value:
> >> "false", data source: default, level: 9 dev/all,
> >> type: bool)
> >> [mishima@node11 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
> >> myprog
> >> [node11.cluster:27895] MCW rank 0 bound to socket 0[core 0[hwt 0]],
socket
> >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >> [node11.cluster:27895] MCW rank 1 bound to socket 1[core 4[hwt 0]],
socket
> >> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >> [node12.cluster:24891] MCW rank 3 bound to socket 1[core 4[hwt 0]],
socket
> >> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >> [node12.cluster:24891] MCW rank 2 bound to socket 0[core 0[hwt 0]],
socket
> >> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >> Hello world from process 0 of 4
> >> Hello world from process 1 of 4
> >> Hello world from process 2 of 4
> >> Hello world from process 3 of 4
> >> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
> >> -hetero-nodes myprog
> >>
--------------------------------------------------------------------------
> >> A request was made to bind to that would result in binding more
> >> processes than cpus on a resource:
> >>
> >> Bind to: CORE
> >> Node: node12
> >> #processes: 2
> >> #cpus: 1
> >>
> >> You can override this protection by adding the "overload-allowed"
> >> option to your binding directive.
> >>
--------------------------------------------------------------------------
> >>
> >>
> >> As far as I checked, data->num_bound seems to become bad in
bind_downwards,
> >> when I put "-hetero-nodes". I hope you can clear the problem.
> >>
> >> Regards,
> >> Tetsuya Mishima
> >>
> >>
> >>> Yes, it's very strange. But I don't think there's any chance that
> >>> I have < 8 actual cores on the node. I guess that you cat replicate
> >>> it with SLURM, please try it again.
> >>>
> >>> I changed to use node10 and node11, then I got the warning against
> >>> node11.
> >>>
> >>> Furthermore, just as an information for you, I tried to add
> >>> "-bind-to core:overload-allowed", then it worked as shown below.
> >>> But I think node11 is never overloaded because it has 8 cores.
> >>>
> >>> qsub: job 8342.manage.cluster completed
> >>> [mishima@manage ~]$ qsub -I -l nodes=node10:ppn=8+node11:ppn=8
> >>> qsub: waiting for job 8343.manage.cluster to start
> >>> qsub: job 8343.manage.cluster ready
> >>>
> >>> [mishima@node10 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>> [mishima@node10 demos]$ cat $PBS_NODEFILE
> >>> node10
> >>> node10
> >>> node10
> >>> node10
> >>> node10
> >>> node10
> >>> node10
> >>> node10
> >>> node11
> >>> node11
> >>> node11
> >>> node11
> >>> node11
> >>> node11
> >>> node11
> >>> node11
> >>> [mishima@node10 demos]$ mpirun -np 4 -cpus-per-proc 4
-report-bindings
> >>> myprog
> >>>
> >>
--------------------------------------------------------------------------
> >>> A request was made to bind to that would result in binding more
> >>> processes than cpus on a resource:
> >>>
> >>> Bind to: CORE
> >>> Node: node11
> >>> #processes: 2
> >>> #cpus: 1
> >>>
> >>> You can override this protection by adding the "overload-allowed"
> >>> option to your binding directive.
> >>>
> >>
--------------------------------------------------------------------------
> >>> [mishima@node10 demos]$ mpirun -np 4 -cpus-per-proc 4
-report-bindings
> >>> -bind-to core:overload-allowed myprog
> >>> [node10.cluster:27020] MCW rank 0 bound to socket 0[core 0[hwt 0]],
> >> socket
> >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >>> [node10.cluster:27020] MCW rank 1 bound to socket 1[core 4[hwt 0]],
> >> socket
> >>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >>> [node11.cluster:26597] MCW rank 3 bound to socket 1[core 4[hwt 0]],
> >> socket
> >>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >>> [node11.cluster:26597] MCW rank 2 bound to socket 0[core 0[hwt 0]],
> >> socket
> >>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >>> Hello world from process 1 of 4
> >>> Hello world from process 0 of 4
> >>> Hello world from process 3 of 4
> >>> Hello world from process 2 of 4
> >>>
> >>> Regards,
> >>> Tetsuya Mishima
> >>>
> >>>
> >>>> Very strange - I can't seem to replicate it. Is there any chance
that
> >> you
> >>> have < 8 actual cores on node12?
> >>>>
> >>>>
> >>>> On Dec 18, 2013, at 4:53 PM, tmish...@jcity.maeda.co.jp wrote:
> >>>>
> >>>>>
> >>>>>
> >>>>> Hi Ralph, sorry for confusing you.
> >>>>>
> >>>>> At that time, I cut and paste the part of "cat $PBS_NODEFILE".
> >>>>> I guess I didn't paste the last line by my mistake.
> >>>>>
> >>>>> I retried the test and below one is exactly what I got when I did
the
> >>> test.
> >>>>>
> >>>>> [mishima@manage ~]$ qsub -I -l nodes=node11:ppn=8+node12:ppn=8
> >>>>> qsub: waiting for job 8338.manage.cluster to start
> >>>>> qsub: job 8338.manage.cluster ready
> >>>>>
> >>>>> [mishima@node11 ~]$ cat $PBS_NODEFILE
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node11
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> node12
> >>>>> [mishima@node11 ~]$ mpirun -np 4 -cpus-per-proc 4 -report-bindings
> >>> myprog
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>> A request was made to bind to that would result in binding more
> >>>>> processes than cpus on a resource:
> >>>>>
> >>>>> Bind to: CORE
> >>>>> Node: node12
> >>>>> #processes: 2
> >>>>> #cpus: 1
> >>>>>
> >>>>> You can override this protection by adding the "overload-allowed"
> >>>>> option to your binding directive.
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>
> >>>>> Regards,
> >>>>>
> >>>>> Tetsuya Mishima
> >>>>>
> >>>>>> I removed the debug in #2 - thanks for reporting it
> >>>>>>
> >>>>>> For #1, it actually looks to me like this is correct. If you look
at
> >>> your
> >>>>> allocation, there are only 7 slots being allocated on node12, yet
you
> >>> have
> >>>>> asked for 8 cpus to be assigned (2 procs with 2
> >>>>>> cpus/proc). So the warning is in fact correct
> >>>>>>
> >>>>>>
> >>>>>> On Dec 18, 2013, at 4:04 PM, tmish...@jcity.maeda.co.jp wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi Ralph, I found that openmpi-1.7.4rc1 was already uploaded. So
> >> I'd
> >>>>> like
> >>>>>>> to report
> >>>>>>> 3 issues mainly regarding -cpus-per-proc.
> >>>>>>>
> >>>>>>> 1) When I use 2 nodes(node11,node12), which has 8 cores each(= 2
> >>>>> sockets X
> >>>>>>> 4 cores/socket),
> >>>>>>> it starts to produce the error again as shown below. At least,
> >>>>>>> openmpi-1.7.4a1r29646 did
> >>>>>>> work well.
> >>>>>>>
> >>>>>>> [mishima@manage ~]$ qsub -I -l nodes=2:ppn=8
> >>>>>>> qsub: waiting for job 8336.manage.cluster to start
> >>>>>>> qsub: job 8336.manage.cluster ready
> >>>>>>>
> >>>>>>> [mishima@node11 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>> [mishima@node11 demos]$ cat $PBS_NODEFILE
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node11
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> node12
> >>>>>>> [mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4
> >>> -report-bindings
> >>>>>>> myprog
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>> A request was made to bind to that would result in binding more
> >>>>>>> processes than cpus on a resource:
> >>>>>>>
> >>>>>>> Bind to: CORE
> >>>>>>> Node: node12
> >>>>>>> #processes: 2
> >>>>>>> #cpus: 1
> >>>>>>>
> >>>>>>> You can override this protection by adding the "overload-allowed"
> >>>>>>> option to your binding directive.
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>
> >>>>>>> Of course it works well using only one node.
> >>>>>>>
> >>>>>>> [mishima@node11 demos]$ mpirun -np 2 -cpus-per-proc 4
> >>> -report-bindings
> >>>>>>> myprog
> >>>>>>> [node11.cluster:26238] MCW rank 0 bound to socket 0[core 0[hwt
0]],
> >>>>> socket
> >>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >>>>>>> [node11.cluster:26238] MCW rank 1 bound to socket 1[core 4[hwt
0]],
> >>>>> socket
> >>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >>>>>>> Hello world from process 1 of 2
> >>>>>>> Hello world from process 0 of 2
> >>>>>>>
> >>>>>>>
> >>>>>>> 2) Adding "-bind-to numa", it works but the message "bind:upward
> >>> target
> >>>>>>> NUMANode type NUMANode" appears.
> >>>>>>> As far as I remember, I didn't see such a kind of message before.
> >>>>>>>
> >>>>>>> mishima@node11 demos]$ mpirun -np 4 -cpus-per-proc 4
> >> -report-bindings
> >>>>>>> -bind-to numa myprog
> >>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
> >> type
> >>>>>>> NUMANode
> >>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
> >> type
> >>>>>>> NUMANode
> >>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
> >> type
> >>>>>>> NUMANode
> >>>>>>> [node11.cluster:26260] [[8844,0],0] bind:upward target NUMANode
> >> type
> >>>>>>> NUMANode
> >>>>>>> [node11.cluster:26260] MCW rank 0 bound to socket 0[core 0[hwt
0]],
> >>>>> socket
> >>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >>>>>>> [node11.cluster:26260] MCW rank 1 bound to socket 1[core 4[hwt
0]],
> >>>>> socket
> >>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >>>>>>> [node12.cluster:23607] MCW rank 3 bound to socket 1[core 4[hwt
0]],
> >>>>> socket
> >>>>>>> 1[core 5[hwt 0]], socket 1[core 6[hwt 0]], so
> >>>>>>> cket 1[core 7[hwt 0]]: [./././.][B/B/B/B]
> >>>>>>> [node12.cluster:23607] MCW rank 2 bound to socket 0[core 0[hwt
0]],
> >>>>> socket
> >>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>> cket 0[core 3[hwt 0]]: [B/B/B/B][./././.]
> >>>>>>> Hello world from process 1 of 4
> >>>>>>> Hello world from process 0 of 4
> >>>>>>> Hello world from process 3 of 4
> >>>>>>> Hello world from process 2 of 4
> >>>>>>>
> >>>>>>>
> >>>>>>> 3) I use PGI compiler. It can not accept compiler switch
> >>>>>>> "-Wno-variadic-macros", which is
> >>>>>>> included in configure script.
> >>>>>>>
> >>>>>>> btl_usnic_CFLAGS="-Wno-variadic-macros"
> >>>>>>>
> >>>>>>> I removed this switch, then I could continue to build 1.7.4rc1.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Tetsuya Mishima
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hmmm...okay, I understand the scenario. Must be something in the
> >>> algo
> >>>>>>> when it only has one node, so it shouldn't be too hard to track
> >> down.
> >>>>>>>>
> >>>>>>>> I'm off on travel for a few days, but will return to this when I
> >> get
> >>>>>>> back.
> >>>>>>>>
> >>>>>>>> Sorry for delay - will try to look at this while I'm gone, but
> >> can't
> >>>>>>> promise anything :-(
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Dec 10, 2013, at 6:58 PM, tmish...@jcity.maeda.co.jp wrote:
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Hi Ralph, sorry for confusing.
> >>>>>>>>>
> >>>>>>>>> We usually logon to "manage", which is our control node.
> >>>>>>>>> From manage, we submit job or enter a remote node such as
> >>>>>>>>> node03 by torque interactive mode(qsub -I).
> >>>>>>>>>
> >>>>>>>>> At that time, instead of torque, I just did rsh to node03 from
> >>> manage
> >>>>>>>>> and ran myprog on the node. I hope you could understand what I
> >> did.
> >>>>>>>>>
> >>>>>>>>> Now, I retried with "-host node03", which still causes the
> >> problem:
> >>>>>>>>> (I comfirmed local run on manage caused the same problem too)
> >>>>>>>>>
> >>>>>>>>> [mishima@manage ~]$ rsh node03
> >>>>>>>>> Last login: Wed Dec 11 11:38:57 from manage
> >>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>>>> [mishima@node03 demos]$
> >>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -host node03
> >> -report-bindings
> >>>>>>>>> -cpus-per-proc 4 -map-by socket myprog
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>> A request was made to bind to that would result in binding more
> >>>>>>>>> processes than cpus on a resource:
> >>>>>>>>>
> >>>>>>>>> Bind to: CORE
> >>>>>>>>> Node: node03
> >>>>>>>>> #processes: 2
> >>>>>>>>> #cpus: 1
> >>>>>>>>>
> >>>>>>>>> You can override this protection by adding the
"overload-allowed"
> >>>>>>>>> option to your binding directive.
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>>
> >>>>>>>>> It' strange, but I have to report that "-map-by socket:span"
> >> worked
> >>>>>>> well.
> >>>>>>>>>
> >>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -host node03
> >> -report-bindings
> >>>>>>>>> -cpus-per-proc 4 -map-by socket:span myprog
> >>>>>>>>> [node03.cluster:11871] MCW rank 2 bound to socket 1[core 8[hwt
> >> 0]],
> >>>>>>> socket
> >>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
> >>>>>>>>> ocket 1[core 11[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 3 bound to socket 1[core 12[hwt
> >>> 0]],
> >>>>>>> socket
> >>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
> >>>>>>>>> socket 1[core 15[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 4 bound to socket 2[core 16[hwt
> >>> 0]],
> >>>>>>> socket>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
> >>>>>>>>> socket 2[core 19[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 5 bound to socket 2[core 20[hwt
> >>> 0]],
> >>>>>>> socket
> >>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
> >>>>>>>>> socket 2[core 23[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 6 bound to socket 3[core 24[hwt
> >>> 0]],
> >>>>>>> socket
> >>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
> >>>>>>>>> socket 3[core 27[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 7 bound to socket 3[core 28[hwt
> >>> 0]],
> >>>>>>> socket
> >>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
> >>>>>>>>> socket 3[core 31[hwt 0]]:
> >>>>>>>>>
> >>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
> >>>>>>>>> [node03.cluster:11871] MCW rank 0 bound to socket 0[core 0[hwt
> >> 0]],
> >>>>>>> socket
> >>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>
> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>> [node03.cluster:11871] MCW rank 1 bound to socket 0[core 4[hwt
> >> 0]],
> >>>>>>> socket
> >>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>
> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>> Hello world from process 3 of 8
> >>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>
> >>>>>>>>> Regards,
> >>>>>>>>> Tetsuya Mishima
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> On Dec 10, 2013, at 6:05 PM, tmish...@jcity.maeda.co.jp wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Ralph,
> >>>>>>>>>>>
> >>>>>>>>>>> I tried again with -cpus-per-proc 2 as shown below.
> >>>>>>>>>>> Here, I found that "-map-by socket:span" worked well.
> >>>>>>>>>>>
> >>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
> >>>>> -cpus-per-proc
> >>>>>>> 2
> >>>>>>>>>>> -map-by socket:span myprog
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 2 bound to socket 1[core 8
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 9[hwt 0]]: [./././././././.][B/B/././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 3 bound to socket 1[core 10
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 11[hwt 0]]: [./././././././.][././B/B
> >>>>>>>>>>> /./././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 4 bound to socket 2[core 16
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 2[core 17[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /./././.][B/B/./././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 5 bound to socket 2[core 18
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 2[core 19[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /./././.][././B/B/./././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 6 bound to socket 3[core 24
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 3[core 25[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /./././.][./././././././.][B/B/./././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 7 bound to socket 3[core 26
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 3[core 27[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /./././.][./././././././.][././B/B/./././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 0 bound to socket 0[core 0
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 1[hwt 0]]: [B/B/./././././.][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10879] MCW rank 1 bound to socket 0[core 2
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 3[hwt 0]]: [././B/B/./././.][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>> Hello world from process 5 of 8> >>>>>>> Hello world from
> >> process 3 of 8
> >>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
> >>>>> -cpus-per-proc
> >>>>>>> 2
> >>>>>>>>>>> -map-by socket myprog
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 2 bound to socket 0[core 4
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 5[hwt 0]]: [././././B/B/./.][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 3 bound to socket 0[core 6
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 7[hwt 0]]: [././././././B/B][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 4 bound to socket 1[core 8
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 9[hwt 0]]: [./././././././.][B/B/././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 5 bound to socket 1[core 10
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 11[hwt 0]]: [./././././././.][././B/B
> >>>>>>>>>>> /./././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 6 bound to socket 1[core 12
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 13[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /B/B/./.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 7 bound to socket 1[core 14
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 15[hwt 0]]: [./././././././.][./././.
> >>>>>>>>>>> /././B/B][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 0 bound to socket 0[core 0
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 1[hwt 0]]: [B/B/./././././.][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:10921] MCW rank 1 bound to socket 0[core 2
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 3[hwt 0]]: [././B/B/./././.][././././.
> >>>>>>>>>>> /././.][./././././././.][./././././././.]
> >>>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>> Hello world from process 3 of 8
> >>>>>>>>>>>
> >>>>>>>>>>> "-np 8" and "-cpus-per-proc 4" just filled all sockets.
> >>>>>>>>>>> In this case, I guess "-map-by socket:span" and "-map-by
> >> socket"
> >>>>> has
> >>>>>>>>> same
> >>>>>>>>>>> meaning.
> >>>>>>>>>>> Therefore, there's no problem about that. Sorry for
distubing.
> >>>>>>>>>>
> >>>>>>>>>> No problem - glad you could clear that up :-)
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> By the way, through this test, I found another problem.
> >>>>>>>>>>> Without torque manager and just using rsh, it causes the same
> >>> error
> >>>>>>>>> like
> >>>>>>>>>>> below:
> >>>>>>>>>>>
> >>>>>>>>>>> [mishima@manage openmpi-1.7]$ rsh node03
> >>>>>>>>>>> Last login: Wed Dec 11 09:42:02 from manage
> >>>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
> >>>>> -cpus-per-proc
> >>>>>>> 4
> >>>>>>>>>>> -map-by socket myprog
> >>>>>>>>>>
> >>>>>>>>>> I don't understand the difference here - you are simply
starting
> >>> it
> >>>>>>> from>>>>> a different node? It looks like everything is expected
to
> >>> run local
> >>>>> to
> >>>>>>>>> mpirun, yes? So there is no rsh actually involved here.
> >>>>>>>>>> Are you still running in an allocation?
> >>>>>>>>>>
> >>>>>>>>>> If you run this with "-host node03" on the cmd line, do you
see
> >>> the
> >>>>>>> same
> >>>>>>>>> problem?
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>>>> A request was made to bind to that would result in binding
more
> >>>>>>>>>>> processes than cpus on a resource:
> >>>>>>>>>>>
> >>>>>>>>>>> Bind to: CORE
> >>>>>>>>>>> Node: node03
> >>>>>>>>>>> #processes: 2
> >>>>>>>>>>> #cpus: 1
> >>>>>>>>>>>
> >>>>>>>>>>> You can override this protection by adding the
> >> "overload-allowed"
> >>>>>>>>>>> option to your binding directive.
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>>>> [mishima@node03 demos]$
> >>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
> >>>>> -cpus-per-proc
> >>>>>>> 4
> >>>>>>>>>>> myprog
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 2 bound to socket 1[core 8
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
> >>>>>>>>>>> ocket 1[core 11[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 3 bound to socket 1[core 12
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
> >>>>>>>>>>> socket 1[core 15[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[./././././././.][././././B/B/B/B][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 4 bound to socket 2[core 16
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
> >>>>>>>>>>> socket 2[core 19[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 5 bound to socket 2[core 20
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
> >>>>>>>>>>> socket 2[core 23[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][././././B/B/B/B][./././././././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 6 bound to socket 3[core 24
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
> >>>>>>>>>>> socket 3[core 27[hwt 0]]:>>>>>
> >>>>>
[./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 7 bound to socket 3[core 28
[hwt
> >>>>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
> >>>>>>>>>>> socket 3[core 31[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][./././././././.][././././B/B/B/B]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 0 bound to socket 0[core 0
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>> [node03.cluster:11036] MCW rank 1 bound to socket 0[core 4
[hwt
> >>> 0]],
> >>>>>>>>> socket
> >>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>>> Hello world from process 3 of 8
> >>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>>
> >>>>>>>>>>> Regards,
> >>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>
> >>>>>>>>>>>> Hmmm...that's strange. I only have 2 sockets on my system,
but
> >>> let
> >>>>>>> me
> >>>>>>>>>>> poke around a bit and see what might be happening.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Dec 10, 2013, at 4:47 PM, tmish...@jcity.maeda.co.jp
wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Ralph,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks. I didn't know the meaning of "socket:span".
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> But it still causes the problem, which seems socket:span
> >>> doesn't
> >>>>>>>>> work.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [mishima@manage demos]$ qsub -I -l nodes=node03:ppn=32
> >>>>>>>>>>>>> qsub: waiting for job 8265.manage.cluster to start
> >>>>>>>>>>>>> qsub: job 8265.manage.cluster ready
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings
> >>>>>>> -cpus-per-proc
> >>>>>>>>> 4
> >>>>>>>>>>>>> -map-by socket:span myprog
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 2 bound to socket 1[core 8
> >> [hwt
> >>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
> >>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 3 bound to socket 1[core 12
> >> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
> >>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 4 bound to socket 2[core 16
> >> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
> >>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 5 bound to socket 2[core 20
> >> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
> >>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 6 bound to socket 3[core 24
> >> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
> >>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 7 bound to socket 3[core 28
> >> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
> >>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 0 bound to socket 0[core 0
> >> [hwt
> >>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>> [node03.cluster:10262] MCW rank 1 bound to socket 0[core 4
> >> [hwt
> >>>>> 0]],
> >>>>>>>>>>> socket
> >>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>>>>
> >>>>>>>
> >> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>> Hello world from process 0 of 8>>>>>>>>>>>>> Hello world
from process 3 of 8
> >>>>>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> No, that is actually correct. We map a socket until full,
> >> then
> >>>>>>> move
> >>>>>>>>> to
> >>>>>>>>>>>>> the next. What you want is --map-by socket:span
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Dec 10, 2013, at 3:42 PM, tmish
i...@jcity.maeda.co.jp
> >> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hi Ralph,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I had a time to try your patch yesterday using
> >>>>>>>>> openmpi-1.7.4a1r29646.
> >>>>>>>>>>>>>>>>>>>>>> It stopped the error but unfortunately "mapping by
> >>>>>>> socket" itself
> >>>>>>>>>>>>> didn't
> >>>>>>>>>>>>>>> work
> >>>>>>>>>>>>>>> well as shown bellow:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [mishima@manage demos]$ qsub -I -l nodes=1:ppn=32
> >>>>>>>>>>>>>>> qsub: waiting for job 8260.manage.cluster to start
> >>>>>>>>>>>>>>> qsub: job 8260.manage.cluster ready
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> [mishima@node04 ~]$ cd ~/Desktop/openmpi-1.7/demos/
> >>>>>>>>>>>>>>> [mishima@node04 demos]$ mpirun -np 8 -report-bindings
> >>>>>>>>> -cpus-per-proc
> >>>>>>>>>>> 4
> >>>>>>>>>>>>>>> -map-by socket myprog
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 2 bound to socket 1[core
8
> >>> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
> >>>>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 3 bound to socket 1[core
12
> >>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
> >>>>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][././././B/B/B/B][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 4 bound to socket 2[core
16
> >>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
> >>>>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 5 bound to socket 2[core
20
> >>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
> >>>>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][./././././././.][././././B/B/B/B][./././././././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 6 bound to socket 3[core
24
> >>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
> >>>>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 7 bound to socket 3[core
28
> >>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
> >>>>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][./././././././.][./././././././.][././././B/B/B/B]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 0 bound to socket 0[core
0
> >>> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> [node04.cluster:27489] MCW rank 1 bound to socket 0[core
4
> >>> [hwt
> >>>>>>> 0]],
> >>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> Hello world from process 2 of 8
> >>>>>>>>>>>>>>> Hello world from process 1 of 8
> >>>>>>>>>>>>>>> Hello world from process 3 of 8
> >>>>>>>>>>>>>>> Hello world from process 0 of 8
> >>>>>>>>>>>>>>> Hello world from process 6 of 8
> >>>>>>>>>>>>>>> Hello world from process 5 of 8
> >>>>>>>>>>>>>>> Hello world from process 4 of 8
> >>>>>>>>>>>>>>> Hello world from process 7 of 8
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I think this should be like this:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> rank 00
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> rank 01
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>> rank 02
> >>>>>>>>>>>>>>>
> >>>>>>>>>
> >>> [./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I fixed this under the trunk (was an issue regardless of
> >> RM)
> >>>>> and
> >>>>>>>>>>> have
> >>>>>>>>>>>>>>> scheduled it for 1.7.4.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>>> Ralph
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> On Nov 25, 2013, at 4:22 PM, tmish...@jcity.maeda.co.jp
> >>> wrote:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Hi Ralph,
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Thank you very much for your quick response.>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> I'm afraid to say that I found one more issuse...
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> It's not so serious. Please check it when you have a
lot
> >> of
> >>>>>>> time.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> The problem is cpus-per-proc with -map-by option under
> >>> Torque
> >>>>>>>>>>>>> manager.
> >>>>>>>>>>>>>>>>> It doesn't work as shown below. I guess you can get the
> >>> same
> >>>>>>>>>>>>>>>>> behaviour under Slurm manager.
> >>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Of course, if I remove -map-by option, it works quite
> >> well.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [mishima@manage testbed2]$ qsub -I -l nodes=1:ppn=32
> >>>>>>>>>>>>>>>>> qsub: waiting for job 8116.manage.cluster to start
> >>>>>>>>>>>>>>>>> qsub: job 8116.manage.cluster ready
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [mishima@node03 ~]$ cd ~/Ducom/testbed2
> >>>>>>>>>>>>>>>>> [mishima@node03 testbed2]$ mpirun -np 8
-report-bindings
> >>>>>>>>>>>>> -cpus-per-proc
> >>>>>>>>>>>>>>> 4
> >>>>>>>>>>>>>>>>> -map-by socket mPre
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>>>>>>>>>> A request was made to bind to that would result in
> >> binding
> >>>>> more
> >>>>>>>>>>>>>>>>> processes than cpus on a resource:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Bind to: CORE
> >>>>>>>>>>>>>>>>> Node: node03>>>>>>> #processes: 2
> >>>>>>>>>>>>>>>>> #cpus: 1
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> You can override this protection by adding the
> >>>>>>> "overload-allowed"
> >>>>>>>>>>>>>>>>> option to your binding directive.
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
--------------------------------------------------------------------------
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> [mishima@node03 testbed2]$ mpirun -np 8
-report-bindings
> >>>>>>>>>>>>> -cpus-per-proc
> >>>>>>>>>>>>>>> 4
> >>>>>>>>>>>>>>>>> mPre
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 2 bound to socket 1
[core
> >> 8
> >>>>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 1[core 9[hwt 0]], socket 1[core 10[hwt 0]], s
> >>>>>>>>>>>>>>>>> ocket 1[core 11[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[./././././././.][B/B/B/B/./././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 3 bound to socket 1
[core
> >> 12
> >>>>> [hwt
> >>>>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 1[core 13[hwt 0]], socket 1[core 14[hwt 0]],
> >>>>>>>>>>>>>>>>> socket 1[core 15[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[./././././././.][././././B/B/B/B][./././././././.][./././././././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 4 bound to socket 2
[core
> >> 16
> >>>>> [hwt
> >>>>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 2[core 17[hwt 0]], socket 2[core 18[hwt 0]],
> >>>>>>>>>>>>>>>>> socket 2[core 19[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][B/B/B/B/./././.][./././././././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 5 bound to socket 2
[core
> >> 20
> >>>>> [hwt
> >>>>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 2[core 21[hwt 0]], socket 2[core 22[hwt 0]],
> >>>>>>>>>>>>>>>>> socket 2[core 23[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][././././B/B/B/B][./././././././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 6 bound to socket 3
[core
> >> 24
> >>>>> [hwt
> >>>>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 3[core 25[hwt 0]], socket 3[core 26[hwt 0]],
> >>>>>>>>>>>>>>>>> socket 3[core 27[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[./././././././.][./././././././.][./././././././.][B/B/B/B/./././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 7 bound to socket 3
[core
> >> 28
> >>>>> [hwt
> >>>>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 3[core 29[hwt 0]], socket 3[core 30[hwt 0]],
> >>>>>>>>>>>>>>>>> socket 3[core 31[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
> >>>
> >>
[./././././././.][./././././././.][./././././././.][././././B/B/B/B]>>>>>>>>>>>>>
> >>
> >>> [node03.cluster:18128] MCW rank 0 bound to socket 0[core 0
> >>>>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so
> >>>>>>>>>>>>>>>>> cket 0[core 3[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[B/B/B/B/./././.][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>>>> [node03.cluster:18128] MCW rank 1 bound to socket 0
[core
> >> 4
> >>>>> [hwt
> >>>>>>>>> 0]],
> >>>>>>>>>>>>>>> socket
> >>>>>>>>>>>>>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so
> >>>>>>>>>>>>>>>>> cket 0[core 7[hwt 0]]:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
[././././B/B/B/B][./././././././.][./././././././.][./././././././.]
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>> Regards,
> >>>>>>>>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Fixed and scheduled to move to 1.7.4. Thanks again!
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Nov 17, 2013, at 6:11 PM, Ralph Castain
> >>>>> <r...@open-mpi.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Thanks! That's precisely where I was going to look
when
> >> I
> >>>>> had
> >>>>>>>>>>>>> time :-)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I'll update tomorrow.
> >>>>>>>>>>>>>>>>>> Ralph
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Sun, Nov 17, 2013 at 7:01 PM,
> >>>>>>>>>>> <tmish...@jcity.maeda.co.jp>wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Hi Ralph,
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> This is the continuous story of "Segmentation fault in
> >>>>>>> oob_tcp.c
> >>>>>>>>>>> of
> >>>>>>>>>>>>>>>>>> openmpi-1.7.4a1r29646".
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I found the cause.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Firstly, I noticed that your hostfile can work and
mine
> >>> can
> >>>>>>> not.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Your host file:
> >>>>>>>>>>>>>>>>>> cat hosts
> >>>>>>>>>>>>>>>>>> bend001 slots=12
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> My host file:
> >>>>>>>>>>>>>>>>>> cat hosts
> >>>>>>>>>>>>>>>>>> node08
> >>>>>>>>>>>>>>>>>> node08
> >>>>>>>>>>>>>>>>>> ...(total 8 lines)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> I modified my script file to add "slots=1" to each
line
> >> of
> >>>>> my
> >>>>>>>>>>>>> hostfile
> >>>>>>>>>>>>>>>>>> just before launching mpirun. Then it worked.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> My host file(modified):
> >>>>>>>>>>>>>>>>>> cat hosts
> >>>>>>>>>>>>>>>>>> node08 slots=1
> >>>>>>>>>>>>>>>>>> node08 slots=1
> >>>>>>>>>>>>>>>>>> ...(total 8 lines)
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Secondary, I confirmed that there's a slight
difference
> >>>>>>> between
> >>>>>>>>>>>>>>>>>> orte/util/hostfile/hostfile.c of 1.7.3 and that of
> >>>>>>>>> 1.7.4a1r29646.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> $ diff
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>
> >> hostfile.c.org ../../../../openmpi-1.7.3/orte/util/hostfile/hostfile.c
> >>>>>>>>>>>>>>>>>> 394,401c394,399
> >>>>>>>>>>>>>>>>>> < if (got_count) {
> >>>>>>>>>>>>>>>>>> < node->slots_given = true;
> >>>>>>>>>>>>>>>>>> < } else if (got_max) {
> >>>>>>>>>>>>>>>>>> < node->slots = node->slots_max;
> >>>>>>>>>>>>>>>>>> < node->slots_given = true;
> >>>>>>>>>>>>>>>>>> < } else {
> >>>>>>>>>>>>>>>>>> < /* should be set by obj_new, but just to be
> >>> clear
> >>>>> */
> >>>>>>>>>>>>>>>>>> < node->slots_given = false;
> >>>>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>>>>> if (!got_count) {
> >>>>>>>>>>>>>>>>>>> if (got_max) {
> >>>>>>>>>>>>>>>>>>> node->slots = node->slots_max;
> >>>>>>>>>>>>>>>>>>> } else {
> >>>>>>>>>>>>>>>>>>> ++node->slots;>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>> ....
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Finally, I added the line 402 below just as a
tentative
> >>>>> trial.
> >>>>>>>>>>>>>>>>>> Then, it worked.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> cat -n orte/util/hostfile/hostfile.c:
> >>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>> 394 if (got_count) {
> >>>>>>>>>>>>>>>>>> 395 node->slots_given = true;
> >>>>>>>>>>>>>>>>>> 396 } else if (got_max) {
> >>>>>>>>>>>>>>>>>> 397 node->slots = node->slots_max;
> >>>>>>>>>>>>>>>>>> 398 node->slots_given = true;
> >>>>>>>>>>>>>>>>>> 399 } else {
> >>>>>>>>>>>>>>>>>> 400 /* should be set by obj_new, but just to
be
> >>>>> clear
> >>>>>>>>> */
> >>>>>>>>>>>>>>>>>> 401 node->slots_given
> >>>>> = false;
> >>>>>>>>>>>>>>>>>> 402 ++node->slots; /* added by tmishima */
> >>>>>>>>>>>>>>>>>> 403 }
> >>>>>>>>>>>>>>>>>> ...
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Please fix the problem properly, because it's just
based
> >>> on
> >>>>> my
> >>>>>>>>>>>>>>>>>> random guess. It's related to the treatment of
hostfile
> >>>>> where
> >>>>>>>>>>> slots
> >>>>>>>>>>>>>>>>>> information is not given.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> Regards,
> >>>>>>>>>>>>>>>>>> Tetsuya Mishima
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>
http://www.open-mpi.org/mailman/listinfo.cgi/users_______________________________________________
> >>
> >>>
> >>>>>
> >>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> users mailing list>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>
users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> users mailing list
> >>>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>> users mailing list
> >>>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> users mailing list
> >>>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> users mailing list
> >>>>>>>>>> us...@open-mpi.org
> >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> users mailing list
> >>>>>>>>> us...@open-mpi.org
> >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> users mailing list
> >>>>>>>> us...@open-mpi.org
> >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> users mailing list
> >>>>>>> us...@open-mpi.org
> >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>>
> >>>>>> _______________________________________________
> >>>>>> users mailing list
> >>>>>> us...@open-mpi.org
> >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>>
> >>>>> _______________________________________________
> >>>>> users mailing list
> >>>>> us...@open-mpi.org
> >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>>
> >>>> _______________________________________________
> >>>> users mailing list
> >>>> us...@open-mpi.org
> >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>
> >>> _______________________________________________
> >>> users mailing list
> >>> us...@open-mpi.org
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>
> >> _______________________________________________
> >> users mailing list
> >> us...@open-mpi.org
> >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
