Wow, that's pretty terrible! :( Is the behavior BTL-specific, perchance? E.G., if you only use certain BTLs, does the delay disappear?
FWIW: the use-all-IP interfaces approach has been in OMPI forever. Sent from my phone. No type good. > On Nov 10, 2014, at 6:42 AM, Reuti <re...@staff.uni-marburg.de> wrote: > >> Am 10.11.2014 um 12:24 schrieb Reuti: >> >> Hi, >> >>> Am 09.11.2014 um 05:38 schrieb Ralph Castain: >>> >>> FWIW: during MPI_Init, each process “publishes” all of its interfaces. Each >>> process receives a complete map of that info for every process in the job. >>> So when the TCP btl sets itself up, it attempts to connect across -all- the >>> interfaces published by the other end. >>> >>> So it doesn’t matter what hostname is provided by the RM. We discover and >>> “share” all of the interface info for every node, and then use them for >>> loadbalancing. >> >> does this lead to any time delay when starting up? I stayed with Open MPI >> 1.6.5 for some time and tried to use Open MPI 1.8.3 now. As there is a delay >> when the applications starts in my first compilation of 1.8.3 I disregarded >> even all my extra options and run it outside of any queuingsystem - the >> delay remains - on two different clusters. > > I forgot to mention: the delay is more or less exactly 2 minutes from the > time I issued `mpiexec` until the `mpihello` starts up (there is no delay for > the initial `ssh` to reach the other node though). > > -- Reuti > > >> I tracked it down, that up to 1.8.1 it is working fine, but 1.8.2 already >> creates this delay when starting up a simple mpihello. I assume it may lay >> in the way how to reach other machines, as with one single machine there is >> no delay. But using one (and only one - no tree spawn involved) additional >> machine already triggers this delay. >> >> Did anyone else notice it? >> >> -- Reuti >> >> >>> HTH >>> Ralph >>> >>> >>>> On Nov 8, 2014, at 8:13 PM, Brock Palen <bro...@umich.edu> wrote: >>>> >>>> Ok I figured, i'm going to have to read some more for my own curiosity. >>>> The reason I mention the Resource Manager we use, and that the hostnames >>>> given but PBS/Torque match the 1gig-e interfaces, i'm curious what path it >>>> would take to get to a peer node when the node list given all match the >>>> 1gig interfaces but yet data is being sent out the 10gig eoib0/ib0 >>>> interfaces. >>>> >>>> I'll go do some measurements and see. >>>> >>>> Brock Palen >>>> www.umich.edu/~brockp >>>> CAEN Advanced Computing >>>> XSEDE Campus Champion >>>> bro...@umich.edu >>>> (734)936-1985 >>>> >>>> >>>> >>>>> On Nov 8, 2014, at 8:30 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> >>>>> wrote: >>>>> >>>>> Ralph is right: OMPI aggressively uses all Ethernet interfaces by >>>>> default. >>>>> >>>>> This short FAQ has links to 2 other FAQs that provide detailed >>>>> information about reachability: >>>>> >>>>> http://www.open-mpi.org/faq/?category=tcp#tcp-multi-network >>>>> >>>>> The usNIC BTL uses UDP for its wire transport and actually does a much >>>>> more standards-conformant peer reachability determination (i.e., it >>>>> actually checks routing tables to see if it can reach a given peer which >>>>> has all kinds of caching benefits, kernel controls if you want them, >>>>> etc.). We haven't back-ported this to the TCP BTL because a) most people >>>>> who use TCP for MPI still use a single L2 address space, and b) no one >>>>> has asked for it. :-) >>>>> >>>>> As for the round robin scheduling, there's no indication from the Linux >>>>> TCP stack what the bandwidth is on a given IP interface. So unless you >>>>> use the btl_tcp_bandwidth_<IP_INTERFACE_NAME> (e.g., >>>>> btl_tcp_bandwidth_eth0) MCA params, OMPI will round-robin across them >>>>> equally. >>>>> >>>>> If you have multiple IP interfaces sharing a single physical link, there >>>>> will likely be no benefit from having Open MPI use more than one of them. >>>>> You should probably use btl_tcp_if_include / btl_tcp_if_exclude to >>>>> select just one. >>>>> >>>>> >>>>> >>>>> >>>>>> On Nov 7, 2014, at 2:53 PM, Brock Palen <bro...@umich.edu> wrote: >>>>>> >>>>>> I was doing a test on our IB based cluster, where I was diabling IB >>>>>> >>>>>> --mca btl ^openib --mca mtl ^mxm >>>>>> >>>>>> I was sending very large messages >1GB and I was surppised by the speed. >>>>>> >>>>>> I noticed then that of all our ethernet interfaces >>>>>> >>>>>> eth0 (1gig-e) >>>>>> ib0 (ip over ib, for lustre configuration at vendor request) >>>>>> eoib0 (ethernet over IB interface for IB -> Ethernet gateway for some >>>>>> extrnal storage support at >1Gig speed >>>>>> >>>>>> I saw all three were getting traffic. >>>>>> >>>>>> We use torque for our Resource Manager and use TM support, the hostnames >>>>>> given by torque match the eth0 interfaces. >>>>>> >>>>>> How does OMPI figure out that it can also talk over the others? How >>>>>> does it chose to load balance? >>>>>> >>>>>> BTW that is fine, but we will use if_exclude on one of the IB ones as >>>>>> ib0 and eoib0 are the same physical device and may screw with load >>>>>> balancing if anyone ver falls back to TCP. >>>>>> >>>>>> Brock Palen >>>>>> www.umich.edu/~brockp >>>>>> CAEN Advanced Computing >>>>>> XSEDE Campus Champion >>>>>> bro...@umich.edu >>>>>> (734)936-1985 >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2014/11/25709.php >>>>> >>>>> >>>>> -- >>>>> Jeff Squyres >>>>> jsquy...@cisco.com >>>>> For corporate legal information go to: >>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2014/11/25713.php >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/11/25715.php >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/11/25716.php >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25721.php > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25722.php
