Hi All, Sorry for the late reply on this. I've been digging through the OpenMPI FAQ. I've never explicitly set the subnet IDs for my IB subnets, so I suspect I'm using the factory defaults. Probably, if I change this, it will "just work". I'll see if the end user is still interested in testing this and, if so, try it out.
Thanks, Tim On Tue, Jun 16, 2015 at 7:15 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > wrote: > Do you have different IB subnet IDs? That would be the only way for Open > MPI to tell the two IB subnets apart. > > > > > On Jun 16, 2015, at 1:25 PM, Tim Miller <btamil...@gmail.com> wrote: > > > > Hi All, > > > > We have a set of nodes which are all connected via InfiniBand, but all > are mutually connected. For example, nodes 1-32 are connected to IB switch > A and 33-64 are connected to switch B, but there is no IB connection > between switches A and B. However, all nodes are mutually routable over TCP. > > > > What we'd like to do is tell OpenMPI to use IB when communicating > amongst nodes 1-32 or 33-64, but to use TCP whenever a node in the set 1-32 > needs to talk to another node in the set 33-64 or vice-versa. We've written > an application in such a way that we can confine most of the bandwidth and > latency sensitive operations to within groups of 32 nodes, but members of > the two groups do have to communicate infrequently via MPI. > > > > Is there any way to tell OpenMPI to use IB within an IB-connected group > and TCP for inter-group communications? Obvoiously, we would need to tell > OpenMPI the membership of the two groups. If there's no such functionality, > would it be a difficult thing to hack in (I'd be glad to give it a try > myself, but I'm not that familiar with the codebase, so a couple of > pointers would be helpful, or a note saying I'm crazy for trying). > > > > Thanks, > > Tim > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27141.php > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/06/27144.php >
