Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Sasso, John (GE Power & Water, Non-GE)]

Thanks again!  I tried --display-devel-map and I think it provides a bit too 
much info for our needs.  However, it is nice to know.

BTW, some interesting behavior in using "--report-bindings --bind-to-core" vs 
"--display-map".  

*  If I use "--report-bindings --bind-to-core" but the MPI tasks on a host fail 
to start up, then nothing is reported.  For example, I had a problem where a 
job started across 4 hosts but the hosts could not communicate with one another 
via TCP/IP.

* If I use "--display-map" then the mapping is shown, even in the failure case 
I mentioned in the last bullet.

* What is nice about "--report-bindings --bind-to-core" over "--display-map" is 
that it will report the binding of each rank to CPU, whereas the latter will 
show you what  ranks are running on a given host.  For our needs, this may be 
sufficient, tho it would be nice to have the CPU bindings shown as well

* If using "--report-bindings --bind-to-core" with OpenMPI 1.4.1 then the 
bindings on just the head node are shown.  In 1.6.1, full bindings across all 
hosts are shown.  (I'd have to read release notes on this...)

--john


-Original Message-
From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
Sent: Thursday, March 27, 2014 7:01 PM
To: Open MPI Users
Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

Oooh...it's Jeff's fault!

Fwiw you can get even more detailed mapping info with --display-devel-map

Sent from my iPhone

> On Mar 27, 2014, at 2:58 PM, "Jeff Squyres (jsquyres)"  
> wrote:
> 
>> On Mar 27, 2014, at 4:06 PM, "Sasso, John (GE Power & Water, Non-GE)" 
>>  wrote:
>> 
>> Yes, I noticed that I could not find --display-map in any of the man pages.  
>> Intentional?
> 
> Oops; nope.  I'll ask Ralph to add it...
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to: 
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)

[Jeff Squyres (jsquyres)]

Good information; thanks.

The short reason for this change in behavior of the affinity options is that 
when we first created affinity (wy back in 1.0 days, no one really cared 
about it much, and so we just did a first attempt).  Gradually over time, 
affinity has become much more important.  As such, we have learned much from 
what our users want and how they want to use affinity.  That has caused a few 
changes in approaches to how we do affinity -- and because our understanding 
has grown, sometime it means that the changes we've made have been 
revolutionary (vs. evolutionary), meaning that CLI options change, behaviors 
change, etc.

Sorry about that -- it really reflects how the whole HPC community is evolving 
its attitude towards affinity over time.

BTW, you should be aware that Open MPI v1.8 -- i.e., the next stable series -- 
is scheduled to be released on Monday.  There's additional changes with regards 
to affinity in 1.8 (compared to the v1.6 series); much of what has been 
discussed on this thread has been in the context of v1.7.x (which is being 
renamed to 1.8 on Monday, per our "feature series eventually turns into stable 
series" versioning philosophy).



On Mar 28, 2014, at 9:47 AM, "Sasso, John (GE Power & Water, Non-GE)" 
 wrote:

> Thanks again!  I tried --display-devel-map and I think it provides a bit too 
> much info for our needs.  However, it is nice to know.
> 
> BTW, some interesting behavior in using "--report-bindings --bind-to-core" vs 
> "--display-map".  
> 
> *  If I use "--report-bindings --bind-to-core" but the MPI tasks on a host 
> fail to start up, then nothing is reported.  For example, I had a problem 
> where a job started across 4 hosts but the hosts could not communicate with 
> one another via TCP/IP.
> 
> * If I use "--display-map" then the mapping is shown, even in the failure 
> case I mentioned in the last bullet.
> 
> * What is nice about "--report-bindings --bind-to-core" over "--display-map" 
> is that it will report the binding of each rank to CPU, whereas the latter 
> will show you what  ranks are running on a given host.  For our needs, this 
> may be sufficient, tho it would be nice to have the CPU bindings shown as well
> 
> * If using "--report-bindings --bind-to-core" with OpenMPI 1.4.1 then the 
> bindings on just the head node are shown.  In 1.6.1, full bindings across all 
> hosts are shown.  (I'd have to read release notes on this...)
> 
> --john
> 
> 
> -Original Message-
> From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Ralph Castain
> Sent: Thursday, March 27, 2014 7:01 PM
> To: Open MPI Users
> Subject: Re: [OMPI users] Mapping ranks to hosts (from MPI error messages)
> 
> Oooh...it's Jeff's fault!
> 
> Fwiw you can get even more detailed mapping info with --display-devel-map
> 
> Sent from my iPhone
> 
>> On Mar 27, 2014, at 2:58 PM, "Jeff Squyres (jsquyres)"  
>> wrote:
>> 
>>> On Mar 27, 2014, at 4:06 PM, "Sasso, John (GE Power & Water, Non-GE)" 
>>>  wrote:
>>> 
>>> Yes, I noticed that I could not find --display-map in any of the man pages. 
>>>  Intentional?
>> 
>> Oops; nope.  I'll ask Ralph to add it...
>> 
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OpenMPI-ROMIO-OrangeFS

[Rob Latham]

On 03/27/2014 06:26 PM, Edgar Gabriel wrote:

I will resubmit a new patch, Rob sent me a pointer to the correct
solution. Its on my to do list for tomorrow/this weekend.


I also found a bad memcopy (i was taking the size of a pointer to a 
thing instead of the size of the thing itself), but that only matters 
if ROMIO uses extended generalized requests.  I trust ticket #1159 is 
still accurate?


https://svn.open-mpi.org/trac/ompi/ticket/1159

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

Re: [OMPI users] OpenMPI-ROMIO-OrangeFS

[Jeff Squyres (jsquyres)]

On Mar 28, 2014, at 12:10 PM, Rob Latham  wrote:

> I also found a bad memcopy (i was taking the size of a pointer to a thing 
> instead of the size of the thing itself), but that only matters if ROMIO uses 
> extended generalized requests.  I trust ticket #1159 is still accurate?
> 
> https://svn.open-mpi.org/trac/ompi/ticket/1159


Yes, it looks up-to-date, meaning that zero work has occurred on it.  :-\

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to: 
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] How to replace --cpus-per-proc by --map-by

[Ralph Castain]

You make a good point, Gus - let me throw the thread open for suggestions on 
how to resolve that problem. We've heard similar concerns raised about other 
features we've added to OMPI over the years, but I'm not sure of the best way 
to communicate such information.

Do we need a better web page, perhaps with some areas dedicated to "features"?

Should we send out "tip of the day" messages to the user list?

Other ideas?

We're open to suggestion, really - just need some help identifying the best way 
to get this info out there.

Thanks!
Ralph

On Mar 27, 2014, at 9:24 AM, Gus Correa  wrote:

> <\begin hijacking this thread>
> 
> I second Saliya's thanks to Tetsuya.
> I've been following this thread, to learn a bit more about
> how to use hardware locality with OpenMPI effectively.
> [I am still using "--bycore"+"--bind-to-core" in most cases,
> and "--cpus-per-proc" occasionally when in hybrid MPI+OpenMP mode.]
> 
> When it comes to hardware locality,
> the syntax and the functionality has changed fast and significantly
> in the recent past.
> Hence, it would be great if the OpenMPI web page could provide pointers
> for the type of external documentation that Tetsuya just sent.
> Perhaps also some additional guidelines and comments
> on what is available on each release/series of OpenMPI,
> and how to use these options.
> 
> There is some material about hwloc,
> but I can't see much about lama ( which means "mud" in my
> first language :) ).
> We can hardly learn things like that from the mpiexec man page
> alone, although it has very good examples.
> 
> Thank you,
> Gus Correa
> 
> <\end hijacking of this thread>
> 
> On 03/27/2014 11:38 AM, Saliya Ekanayake wrote:
>> Thank you, this is really helpful.
>> 
>> Saliya
>> 
>> 
>> On Thu, Mar 27, 2014 at 5:11 AM, > > wrote:
>> 
>> 
>> 
>>Mapping and binding is related to so called process affinity.
>>It's a bit difficult for me to explain ...
>> 
>>So please see this URL below(especially the first half part
>>of it - from 1 to 20 pages):
>>
>> http://www.slideshare.net/jsquyres/open-mpi-explorations-in-process-affinity-eurompi13-presentation
>> 
>>Although these slides by Jeff are the explanation for LAMA,
>>which is another mapping system installed in the openmpi-1.7
>>series, I guess you can easily understand what is mapping and
>>binding in general terms.
>> 
>>Tetsuya
>> 
>> > Thank you Tetsuya - it worked.
>> >
>> > Btw. what's the difference between mapping and binding? I think I
>>am bit
>>confused here.
>> >
>> > Thank you,
>> > Saliya
>> >
>> >
>> > On Thu, Mar 27, 2014 at 4:19 AM,  >>wrote:
>> >
>> >
>> > Hi Saliya,
>> >
>> > What you want to do is map-by node. So please try below:
>> >
>> > -np 2 --map-by node:pe=4 --bind-to core
>> >
>> > You might not need to add --bind-to core, because it's default
>>binding.
>> >
>> > Tetsuya
>> >
>> > > Hi,
>> > >
>> > > I see in v.1.7.5rc5 --cpus-per-proc is deprecated and is advised to
>> > replace by --map-by :PE=N.
>> > > I've tried this but I couldn't get the expected allocation of
>>procs.
>> > >
>> > > For example I was running 2 procs on 2 nodes each with 2
>>sockets where
>>a
>> > socket has 4 cores. I wanted 1 proc per node and bound to all
>>cores in
>>one
>> > of the sockets. I could get this by using
>> > >
>> > > --bind-to core: --map-by ppr:1:node --cpus-per-proc 4 -np 2
>> > >
>> > > Then it'll show bindings as
>> > >
>> > > [i51:32274] MCW rank 0 bound to socket 0[core 0[hwt 0]], socket
>>0[core
>>1
>> > [hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]:
>> > [B/B/B/B][./././.]
>> > > [i52:31765] MCW rank 1 bound to socket 0[core 0[hwt 0]], socket
>>0[core
>>1
>> > [hwt 0]], socket 0[core 2[hwt 0]], socket 0[core 3[hwt 0]]:
>> > [B/B/B/B][./././.]
>> > >
>> > >
>> > > Is there a better way without using -cpus-per-proc as suggested
>>to get
>> > the same effect?
>> > >
>> > > Thank you,
>> > > Saliya
>> > >
>> > >
>> > >
>> > > --
>> > > Saliya Ekanayake esal...@gmail.com 
>> > > Cell 812-391-4914  Home 812-961-6383
>>
>> > > http://saliya.org___
>> > > users mailing list
>> > >
>>users@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>> >
>> > ___
>> > users mailing list
>> > us...@open-mpi.org 
>> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>> >
>> >
>> >
>> > --
>> > Saliya Ekanayake esal...@gmail.com