Am 10.11.2014 um 12:50 schrieb Jeff Squyres (jsquyres): > Wow, that's pretty terrible! :( > > Is the behavior BTL-specific, perchance? E.G., if you only use certain BTLs, > does the delay disappear?
You mean something like: reuti@annemarie:~> date; mpiexec -mca btl self,tcp -n 4 --hostfile machines ./mpihello; date Mon Nov 10 13:44:34 CET 2014 Hello World from Node 1. Total: 4 Universe: 4 Hello World from Node 0. Hello World from Node 3. Hello World from Node 2. Mon Nov 10 13:46:42 CET 2014 (the above was even the latest v1.8.3-186-g978f61d) Falling back to 1.8.1 gives (as expected): reuti@annemarie:~> date; mpiexec -mca btl self,tcp -n 4 --hostfile machines ./mpihello; date Mon Nov 10 13:49:51 CET 2014 Hello World from Node 1. Total: 4 Universe: 4 Hello World from Node 0. Hello World from Node 2. Hello World from Node 3. Mon Nov 10 13:49:53 CET 2014 -- Reuti > FWIW: the use-all-IP interfaces approach has been in OMPI forever. > > Sent from my phone. No type good. > >> On Nov 10, 2014, at 6:42 AM, Reuti <re...@staff.uni-marburg.de> wrote: >> >>> Am 10.11.2014 um 12:24 schrieb Reuti: >>> >>> Hi, >>> >>>> Am 09.11.2014 um 05:38 schrieb Ralph Castain: >>>> >>>> FWIW: during MPI_Init, each process “publishes” all of its interfaces. >>>> Each process receives a complete map of that info for every process in the >>>> job. So when the TCP btl sets itself up, it attempts to connect across >>>> -all- the interfaces published by the other end. >>>> >>>> So it doesn’t matter what hostname is provided by the RM. We discover and >>>> “share” all of the interface info for every node, and then use them for >>>> loadbalancing. >>> >>> does this lead to any time delay when starting up? I stayed with Open MPI >>> 1.6.5 for some time and tried to use Open MPI 1.8.3 now. As there is a >>> delay when the applications starts in my first compilation of 1.8.3 I >>> disregarded even all my extra options and run it outside of any >>> queuingsystem - the delay remains - on two different clusters. >> >> I forgot to mention: the delay is more or less exactly 2 minutes from the >> time I issued `mpiexec` until the `mpihello` starts up (there is no delay >> for the initial `ssh` to reach the other node though). >> >> -- Reuti >> >> >>> I tracked it down, that up to 1.8.1 it is working fine, but 1.8.2 already >>> creates this delay when starting up a simple mpihello. I assume it may lay >>> in the way how to reach other machines, as with one single machine there is >>> no delay. But using one (and only one - no tree spawn involved) additional >>> machine already triggers this delay. >>> >>> Did anyone else notice it? >>> >>> -- Reuti >>> >>> >>>> HTH >>>> Ralph >>>> >>>> >>>>> On Nov 8, 2014, at 8:13 PM, Brock Palen <bro...@umich.edu> wrote: >>>>> >>>>> Ok I figured, i'm going to have to read some more for my own curiosity. >>>>> The reason I mention the Resource Manager we use, and that the hostnames >>>>> given but PBS/Torque match the 1gig-e interfaces, i'm curious what path >>>>> it would take to get to a peer node when the node list given all match >>>>> the 1gig interfaces but yet data is being sent out the 10gig eoib0/ib0 >>>>> interfaces. >>>>> >>>>> I'll go do some measurements and see. >>>>> >>>>> Brock Palen >>>>> www.umich.edu/~brockp >>>>> CAEN Advanced Computing >>>>> XSEDE Campus Champion >>>>> bro...@umich.edu >>>>> (734)936-1985 >>>>> >>>>> >>>>> >>>>>> On Nov 8, 2014, at 8:30 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> >>>>>> wrote: >>>>>> >>>>>> Ralph is right: OMPI aggressively uses all Ethernet interfaces by >>>>>> default. >>>>>> >>>>>> This short FAQ has links to 2 other FAQs that provide detailed >>>>>> information about reachability: >>>>>> >>>>>> http://www.open-mpi.org/faq/?category=tcp#tcp-multi-network >>>>>> >>>>>> The usNIC BTL uses UDP for its wire transport and actually does a much >>>>>> more standards-conformant peer reachability determination (i.e., it >>>>>> actually checks routing tables to see if it can reach a given peer which >>>>>> has all kinds of caching benefits, kernel controls if you want them, >>>>>> etc.). We haven't back-ported this to the TCP BTL because a) most >>>>>> people who use TCP for MPI still use a single L2 address space, and b) >>>>>> no one has asked for it. :-) >>>>>> >>>>>> As for the round robin scheduling, there's no indication from the Linux >>>>>> TCP stack what the bandwidth is on a given IP interface. So unless you >>>>>> use the btl_tcp_bandwidth_<IP_INTERFACE_NAME> (e.g., >>>>>> btl_tcp_bandwidth_eth0) MCA params, OMPI will round-robin across them >>>>>> equally. >>>>>> >>>>>> If you have multiple IP interfaces sharing a single physical link, there >>>>>> will likely be no benefit from having Open MPI use more than one of >>>>>> them. You should probably use btl_tcp_if_include / btl_tcp_if_exclude >>>>>> to select just one. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Nov 7, 2014, at 2:53 PM, Brock Palen <bro...@umich.edu> wrote: >>>>>>> >>>>>>> I was doing a test on our IB based cluster, where I was diabling IB >>>>>>> >>>>>>> --mca btl ^openib --mca mtl ^mxm >>>>>>> >>>>>>> I was sending very large messages >1GB and I was surppised by the >>>>>>> speed. >>>>>>> >>>>>>> I noticed then that of all our ethernet interfaces >>>>>>> >>>>>>> eth0 (1gig-e) >>>>>>> ib0 (ip over ib, for lustre configuration at vendor request) >>>>>>> eoib0 (ethernet over IB interface for IB -> Ethernet gateway for some >>>>>>> extrnal storage support at >1Gig speed >>>>>>> >>>>>>> I saw all three were getting traffic. >>>>>>> >>>>>>> We use torque for our Resource Manager and use TM support, the >>>>>>> hostnames given by torque match the eth0 interfaces. >>>>>>> >>>>>>> How does OMPI figure out that it can also talk over the others? How >>>>>>> does it chose to load balance? >>>>>>> >>>>>>> BTW that is fine, but we will use if_exclude on one of the IB ones as >>>>>>> ib0 and eoib0 are the same physical device and may screw with load >>>>>>> balancing if anyone ver falls back to TCP. >>>>>>> >>>>>>> Brock Palen >>>>>>> www.umich.edu/~brockp >>>>>>> CAEN Advanced Computing >>>>>>> XSEDE Campus Champion >>>>>>> bro...@umich.edu >>>>>>> (734)936-1985 >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> Link to this post: >>>>>>> http://www.open-mpi.org/community/lists/users/2014/11/25709.php >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Squyres >>>>>> jsquy...@cisco.com >>>>>> For corporate legal information go to: >>>>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> Link to this post: >>>>>> http://www.open-mpi.org/community/lists/users/2014/11/25713.php >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2014/11/25715.php >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/11/25716.php >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/11/25721.php >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25722.php > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25724.php >
