Hi All, We have a set of nodes which are all connected via InfiniBand, but all are mutually connected. For example, nodes 1-32 are connected to IB switch A and 33-64 are connected to switch B, but there is no IB connection between switches A and B. However, all nodes are mutually routable over TCP.
What we'd like to do is tell OpenMPI to use IB when communicating amongst nodes 1-32 or 33-64, but to use TCP whenever a node in the set 1-32 needs to talk to another node in the set 33-64 or vice-versa. We've written an application in such a way that we can confine most of the bandwidth and latency sensitive operations to within groups of 32 nodes, but members of the two groups do have to communicate infrequently via MPI. Is there any way to tell OpenMPI to use IB within an IB-connected group and TCP for inter-group communications? Obvoiously, we would need to tell OpenMPI the membership of the two groups. If there's no such functionality, would it be a difficult thing to hack in (I'd be glad to give it a try myself, but I'm not that familiar with the codebase, so a couple of pointers would be helpful, or a note saying I'm crazy for trying). Thanks, Tim
