On Mar 16, 2014, at 12:16 AM, Jianyu Liu <jerry_...@msn.com> wrote: > Thanks for your kindly input > > More further questions > > 1. How to check and change the ordering of network interfaces, such as tcp, > ib, etc., in the kernel?
Heck if I know - does it matter? If you want to use a specific one, just tell it to do so. > > 2. One of my applications only can run with "--mca btl tcp,self", otherwise > aborted without specific error messages, even if run on single node. How > to figure out the possible reasons? Sounds odd - if it was an MPI error, it should say something. You can configure it with "--enable-debug" to add debug capability, and then do things like add "-mca plm_base_verbose 5" to watch the launch sequence, and "-mca odls_base_verbose 5" to see the procs actually spawned. Setting "-mca btl_base_verbose 100" should tell you something about what is going on in the btl layer. Etc. > > Appreciating your kindly input > > Jianyu > > > > From: r...@open-mpi.org > > Date: Sat, 15 Mar 2014 07:21:31 -0700 > > To: us...@open-mpi.org > > Subject: Re: [OMPI users] Question about '--mca btl tcp,self' > > > > > > On Mar 14, 2014, at 10:18 PM, Jianyu Liu <jerry_...@msn.com> wrote: > > > > >> On Mar 14, 2014, at 10:16:34 AM,Jeff Squyres <jsquyres_at_[hidden]> > > >> wrote: > > >> > > >>> On Mar 14, 2014, at 10:11 AM, Ralph Castain <rhc_at_[hidden]> wrote: > > >>> > > >>>> 1. If specified '--mca btl tcp,self', which interface application will > > >>>> run on, use GigE adaper OR use the OpenFabrics interface in IP over IB > > >>>> mode (just like a high performance GigE adapter) ? > > >>> > > >>> Both - ip over ib looks just like an Ethernet adaptor > > >> > > >> > > >> To be clear: the TCP BTL will use all TCP interfaces (regardless of > > >> underlying physical transport). Your GigE adapter and your IP adapter > > >> both present IP interfaces to>the OS, and both support TCP. So the TCP > > >> BTL will use them, because it just sees the TCP/IP interfaces. > > > > > > Thanks for your kindly input. > > > > > > Please see if I have understood correctly > > > > > > Assume there are two nework > > > Gigabit Ethernet > > > > > > eth0-renamed : 192.168.[1-22].[1-14] / 255.255.192.0 > > > > > > InfiniBand network > > > > > > ib0 : 172.20.[1-22].[1-4] / 255.255.0.0 > > > > > > > > > 1. If specified '--mca btl tcp,self > > > > > > The control information ( such as setup and teardown ) are routed to and > > > passed by Gigabit Ethernet in TCP/IP mode > > > > Not necessarily - the out-of-band (OOB) system will pickup one of the TCP > > interfaces, but which one depends on the ordering in the kernel. > > > > > The MPI messages are routed to and passed by InfiniBand network in IP > > > over IB mode > > > > Not necessarily - could use either device > > > > > On the same machine, the TCP lookback device will be used for passing > > > control and MPI messages > > > > I believe the TCP BTL would use the selected device for loopback, ignoring > > the loopback device > > > > > > > > 2. If specified '--mca btl tcp,self --mca btl_tcp_if_include ib0' > > > > > > Both of control information ( such as setup and teardown ) and MPI > > > messages are routed to and passed by InfiniBand network in IP over IB mode > > > > No - control info is sent by the OOB, not the BTL. To get what you > > describe, you would have to add "-mca oob_tcp_if_include ib0" > > > > > On the same machine, The TCP lookback device will be used for passing > > > control and MPI messages > > > > No - the TCP MPI messages would loopback via the ib0 device > > > > > > > > > > > 3. If specified '--mca btl openib,self' > > > > > > The control information ( such as setup and teardown ) are routed to and > > > passed by InfiniBand network in IP over IB mode > > > > Not necessarily - same answer as #1 > > > > > The MPI messages are routed to and passed by InfiniBand network in RDMA > > > mode > > > > Well, it will use IB, but may not use RDMA. That is an internal decision > > tree made per-message based on a variety of factors > > > > > On the same machine, the TCP lookback device will be used for passing > > > control and MPI messages > > > > No - you excluded TCP for MPI messages, and so it would have to loopback > > within the IB stack. Control messages would loopback via TCP > > > > > > > > > > > 4. If without specifiying any 'mca btl' parameters > > > > > > The control information ( such as setup and teardown ) are routed to and > > > passed by Gigabit Ethernet in TCP/IP mode > > > > Not necessarily - same answer as #1 > > > > > The MPI messages are routed and passed by InfiniBand network in RDMA mode > > > > Same as #3 > > > > > On the same machine, the shared memory (sm) BTL will be used for control > > > and MPI passing messages > > > > Not for control - just for MPI > > > > > > > > > > > Appreciating your kindly input > > > > > > Jianyu > > > _______________________________________________ > > > users mailing list > > > us...@open-mpi.org > > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
