Am 10.11.2014 um 12:24 schrieb Reuti: > Hi, > > Am 09.11.2014 um 05:38 schrieb Ralph Castain: > >> FWIW: during MPI_Init, each process “publishes” all of its interfaces. Each >> process receives a complete map of that info for every process in the job. >> So when the TCP btl sets itself up, it attempts to connect across -all- the >> interfaces published by the other end. >> >> So it doesn’t matter what hostname is provided by the RM. We discover and >> “share” all of the interface info for every node, and then use them for >> loadbalancing. > > does this lead to any time delay when starting up? I stayed with Open MPI > 1.6.5 for some time and tried to use Open MPI 1.8.3 now. As there is a delay > when the applications starts in my first compilation of 1.8.3 I disregarded > even all my extra options and run it outside of any queuingsystem - the delay > remains - on two different clusters.
I forgot to mention: the delay is more or less exactly 2 minutes from the time I issued `mpiexec` until the `mpihello` starts up (there is no delay for the initial `ssh` to reach the other node though). -- Reuti > I tracked it down, that up to 1.8.1 it is working fine, but 1.8.2 already > creates this delay when starting up a simple mpihello. I assume it may lay in > the way how to reach other machines, as with one single machine there is no > delay. But using one (and only one - no tree spawn involved) additional > machine already triggers this delay. > > Did anyone else notice it? > > -- Reuti > > >> HTH >> Ralph >> >> >>> On Nov 8, 2014, at 8:13 PM, Brock Palen <bro...@umich.edu> wrote: >>> >>> Ok I figured, i'm going to have to read some more for my own curiosity. The >>> reason I mention the Resource Manager we use, and that the hostnames given >>> but PBS/Torque match the 1gig-e interfaces, i'm curious what path it would >>> take to get to a peer node when the node list given all match the 1gig >>> interfaces but yet data is being sent out the 10gig eoib0/ib0 interfaces. >>> >>> I'll go do some measurements and see. >>> >>> Brock Palen >>> www.umich.edu/~brockp >>> CAEN Advanced Computing >>> XSEDE Campus Champion >>> bro...@umich.edu >>> (734)936-1985 >>> >>> >>> >>>> On Nov 8, 2014, at 8:30 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> >>>> wrote: >>>> >>>> Ralph is right: OMPI aggressively uses all Ethernet interfaces by default. >>>> >>>> >>>> This short FAQ has links to 2 other FAQs that provide detailed information >>>> about reachability: >>>> >>>> http://www.open-mpi.org/faq/?category=tcp#tcp-multi-network >>>> >>>> The usNIC BTL uses UDP for its wire transport and actually does a much >>>> more standards-conformant peer reachability determination (i.e., it >>>> actually checks routing tables to see if it can reach a given peer which >>>> has all kinds of caching benefits, kernel controls if you want them, >>>> etc.). We haven't back-ported this to the TCP BTL because a) most people >>>> who use TCP for MPI still use a single L2 address space, and b) no one has >>>> asked for it. :-) >>>> >>>> As for the round robin scheduling, there's no indication from the Linux >>>> TCP stack what the bandwidth is on a given IP interface. So unless you >>>> use the btl_tcp_bandwidth_<IP_INTERFACE_NAME> (e.g., >>>> btl_tcp_bandwidth_eth0) MCA params, OMPI will round-robin across them >>>> equally. >>>> >>>> If you have multiple IP interfaces sharing a single physical link, there >>>> will likely be no benefit from having Open MPI use more than one of them. >>>> You should probably use btl_tcp_if_include / btl_tcp_if_exclude to select >>>> just one. >>>> >>>> >>>> >>>> >>>> On Nov 7, 2014, at 2:53 PM, Brock Palen <bro...@umich.edu> wrote: >>>> >>>>> I was doing a test on our IB based cluster, where I was diabling IB >>>>> >>>>> --mca btl ^openib --mca mtl ^mxm >>>>> >>>>> I was sending very large messages >1GB and I was surppised by the speed. >>>>> >>>>> I noticed then that of all our ethernet interfaces >>>>> >>>>> eth0 (1gig-e) >>>>> ib0 (ip over ib, for lustre configuration at vendor request) >>>>> eoib0 (ethernet over IB interface for IB -> Ethernet gateway for some >>>>> extrnal storage support at >1Gig speed >>>>> >>>>> I saw all three were getting traffic. >>>>> >>>>> We use torque for our Resource Manager and use TM support, the hostnames >>>>> given by torque match the eth0 interfaces. >>>>> >>>>> How does OMPI figure out that it can also talk over the others? How does >>>>> it chose to load balance? >>>>> >>>>> BTW that is fine, but we will use if_exclude on one of the IB ones as ib0 >>>>> and eoib0 are the same physical device and may screw with load balancing >>>>> if anyone ver falls back to TCP. >>>>> >>>>> Brock Palen >>>>> www.umich.edu/~brockp >>>>> CAEN Advanced Computing >>>>> XSEDE Campus Champion >>>>> bro...@umich.edu >>>>> (734)936-1985 >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> Link to this post: >>>>> http://www.open-mpi.org/community/lists/users/2014/11/25709.php >>>> >>>> >>>> -- >>>> Jeff Squyres >>>> jsquy...@cisco.com >>>> For corporate legal information go to: >>>> http://www.cisco.com/web/about/doing_business/legal/cri/ >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> Link to this post: >>>> http://www.open-mpi.org/community/lists/users/2014/11/25713.php >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>> Link to this post: >>> http://www.open-mpi.org/community/lists/users/2014/11/25715.php >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2014/11/25716.php >> > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2014/11/25721.php >
