Re: [OMPI users] problem using mpirun over multiple nodes
Hi Jeff, I was wondering how I can check whether there is any firewall software . In fact I can use ssh to go from one machine to another . But, only with mpirun , it does not work. I was wondering whether it is possible that even in presence of firewall ssh may work but mpirun may not. Jagannath On Wed, May 25, 2011 at 10:42 PM, Jeff Squyres (jsquyres) < jsquy...@cisco.com> wrote: > Are you running any firewall software? > > Sent from my phone. No type good. > > On May 25, 2011, at 10:41 PM, "Jagannath Mondal" < > jagannath.mon...@gmail.com> wrote: > > Hi, > I am having a problem in running mpirun over multiple nodes. > To run a job over two 8-core processors, I generated a hostfile as > follows: > yethiraj30 slots=8 max_slots=8 > yethiraj31 slots=8 max_slots=8 > > These two machines are intra-connected and I have installed openmpi 1.3.3. > Then If I try to run the replica exchange simulation using the following > command: > mpirun -np 16 --hostfile hostfile mdrun_4mpi -s topol_.tpr -multi 16 > -replex 100 >& log_replica_test > > But I find following error and job does not proceed at all : > btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] connect() to > 192.168.0.31 failed: No route to host (113) > > Here is the full details: > > NNODES=16, MYRANK=0, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=1, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=4, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=2, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=6, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=3, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=5, HOSTNAME=yethiraj30 > NNODES=16, MYRANK=7, HOSTNAME=yethiraj30 > [yethiraj30][[22604,1],0][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > [yethiraj30][[22604,1],4][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > [yethiraj30][[22604,1],6][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > [yethiraj30][[22604,1],1][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > [yethiraj30][[22604,1],3][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > [yethiraj30][[22604,1],2][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > NNODES=16, MYRANK=10, HOSTNAME=yethiraj31 > NNODES=16, MYRANK=12, HOSTNAME=yethiraj31 > > I am not sure how to resolve this issue. In general, I can go from one > machine to another without any problem using ssh. But, when I am trying to > run openmpi over both the machines, I get this error. Any help will be > appreciated. > > Jagannath > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] MPI_COMM_DUP freeze with OpenMPI 1.4.1
Jeff Squyres wrote: On May 24, 2011, at 4:42 AM, francoise.r...@obs.ujf-grenoble.fr wrote: CALL MPI_COMM_SIZE(id%COMM, id%NPROCS, IERR ) IF ( id%PAR .eq. 0 ) THEN IF ( id%MYID .eq. MASTER ) THEN color = MPI_UNDEFINED ELSE color = 0 END IF CALL MPI_COMM_SPLIT( id%COMM, color, 0, id%COMM_NODES, IERR ) id%NSLAVES = id%NPROCS - 1 ELSE CALL MPI_COMM_DUP( id%COMM, id%COMM_NODES, IERR ) id%NSLAVES = id%NPROCS END IF IF (id%PAR .ne. 0 .or. id%MYID .NE. MASTER) THEN CALL MPI_COMM_DUP( id%COMM_NODES, id%COMM_LOAD, IERR ENDIF Actually, we look at the first case, that is id%par = 0. But the MPI_COMM_SPLIT routine is called by all the processes and creates a new communicator named "id%COMM_NODES". This communicator contains all the slaves, but not the master. The first MPI_COMM_DUP is not executed, the second one is executed on all the slaves nodes (id%MYID .NE. MASTER ), because the communicator is "id%COMM_NODES" and so implies all the processes of this communicator. Hmm. Are you sure that id%myid is relative to id%comm? I don't see its assignment in your code snipit. Yes, id%myid is relative to id%comm. It is assigned, just before in the code, by all the processes, by the following call : CALL MPI_COMM_RANK(id%COMM, id%MYID, IERR)
Re: [OMPI users] MPI_COMM_DUP freeze with OpenMPI 1.4.1
On May 26, 2011, at 4:43 AM, francoise.r...@obs.ujf-grenoble.fr wrote: CALL MPI_COMM_SIZE(id%COMM, id%NPROCS, IERR ) IF ( id%PAR .eq. 0 ) THEN IF ( id%MYID .eq. MASTER ) THEN color = MPI_UNDEFINED ELSE color = 0 END IF CALL MPI_COMM_SPLIT( id%COMM, color, 0, id%COMM_NODES, IERR ) id%NSLAVES = id%NPROCS - 1 ELSE CALL MPI_COMM_DUP( id%COMM, id%COMM_NODES, IERR ) id%NSLAVES = id%NPROCS END IF IF (id%PAR .ne. 0 .or. id%MYID .NE. MASTER) THEN CALL MPI_COMM_DUP( id%COMM_NODES, id%COMM_LOAD, IERR ENDIF > Yes, id%myid is relative to id%comm. It is assigned, just before in the code, > by all the processes, by the following call : > CALL MPI_COMM_RANK(id%COMM, id%MYID, IERR) I'm out of ideas. :-( Can you create a short reproducer code? -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] problem using mpirun over multiple nodes
ssh may be allowed but other random TCP ports may not. iptables is the typical firewall software that most Linux installations use; it may have been enabled by default. I'm a little doubtful that this is your problem, though, because you're apparently able to *launch* your application, which means that OMPI's out-of-band communication system was able to make some sockets. So it's a little weird that the MPI layer's TCP sockets were borked. But let's check for firewall software, first... On May 26, 2011, at 12:42 AM, Jagannath Mondal wrote: > Hi Jeff, > I was wondering how I can check whether there is any firewall software . > In fact I can use ssh to go from one machine to another . But, only with > mpirun , it does not work. I was wondering whether it is possible that even > in presence of firewall ssh may work but mpirun may not. > Jagannath > > On Wed, May 25, 2011 at 10:42 PM, Jeff Squyres (jsquyres) > wrote: > Are you running any firewall software? > > Sent from my phone. No type good. > > On May 25, 2011, at 10:41 PM, "Jagannath Mondal" > wrote: > >> Hi, >> I am having a problem in running mpirun over multiple nodes. >> To run a job over two 8-core processors, I generated a hostfile as follows: >> yethiraj30 slots=8 max_slots=8 >> yethiraj31 slots=8 max_slots=8 >> >> These two machines are intra-connected and I have installed openmpi 1.3.3. >> Then If I try to run the replica exchange simulation using the following >> command: >> mpirun -np 16 --hostfile hostfile mdrun_4mpi -s topol_.tpr -multi 16 >> -replex 100 >& log_replica_test >> >> But I find following error and job does not proceed at all : >> btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] connect() to >> 192.168.0.31 failed: No route to host (113) >> >> Here is the full details: >> >> NNODES=16, MYRANK=0, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=1, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=4, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=2, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=6, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=3, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=5, HOSTNAME=yethiraj30 >> NNODES=16, MYRANK=7, HOSTNAME=yethiraj30 >> [yethiraj30][[22604,1],0][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> [yethiraj30][[22604,1],4][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> [yethiraj30][[22604,1],6][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> [yethiraj30][[22604,1],1][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> [yethiraj30][[22604,1],3][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> [yethiraj30][[22604,1],2][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] >> connect() to 192.168.0.31 failed: No route to host (113) >> NNODES=16, MYRANK=10, HOSTNAME=yethiraj31 >> NNODES=16, MYRANK=12, HOSTNAME=yethiraj31 >> >> I am not sure how to resolve this issue. In general, I can go from one >> machine to another without any problem using ssh. But, when I am trying to >> run openmpi over both the machines, I get this error. Any help will be >> appreciated. >> >> Jagannath >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI users] problem using mpirun over multiple nodes
Hi Jeff, Thanks to you, I figured the problem . As you suspected, it was iptables which was acting as firewalls in some machines. So, after I stopped the iptable, the MPI communication is going fine. Even I tried with 5 machines together and the communication is going allright. Thanks again, Jagannath On Thu, May 26, 2011 at 5:19 AM, Jeff Squyres wrote: > ssh may be allowed but other random TCP ports may not. > > iptables is the typical firewall software that most Linux installations > use; it may have been enabled by default. > > I'm a little doubtful that this is your problem, though, because you're > apparently able to *launch* your application, which means that OMPI's > out-of-band communication system was able to make some sockets. So it's a > little weird that the MPI layer's TCP sockets were borked. But let's check > for firewall software, first... > > > On May 26, 2011, at 12:42 AM, Jagannath Mondal wrote: > > > Hi Jeff, > > I was wondering how I can check whether there is any firewall > software . In fact I can use ssh to go from one machine to another . But, > only with mpirun , it does not work. I was wondering whether it is possible > that even in presence of firewall ssh may work but mpirun may not. > > Jagannath > > > > On Wed, May 25, 2011 at 10:42 PM, Jeff Squyres (jsquyres) < > jsquy...@cisco.com> wrote: > > Are you running any firewall software? > > > > Sent from my phone. No type good. > > > > On May 25, 2011, at 10:41 PM, "Jagannath Mondal" < > jagannath.mon...@gmail.com> wrote: > > > >> Hi, > >> I am having a problem in running mpirun over multiple nodes. > >> To run a job over two 8-core processors, I generated a hostfile as > follows: > >> yethiraj30 slots=8 max_slots=8 > >> yethiraj31 slots=8 max_slots=8 > >> > >> These two machines are intra-connected and I have installed openmpi > 1.3.3. > >> Then If I try to run the replica exchange simulation using the following > command: > >> mpirun -np 16 --hostfile hostfile mdrun_4mpi -s topol_.tpr -multi 16 > -replex 100 >& log_replica_test > >> > >> But I find following error and job does not proceed at all : > >> btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] connect() > to 192.168.0.31 failed: No route to host (113) > >> > >> Here is the full details: > >> > >> NNODES=16, MYRANK=0, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=1, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=4, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=2, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=6, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=3, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=5, HOSTNAME=yethiraj30 > >> NNODES=16, MYRANK=7, HOSTNAME=yethiraj30 > >> > [yethiraj30][[22604,1],0][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> > [yethiraj30][[22604,1],4][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> > [yethiraj30][[22604,1],6][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> > [yethiraj30][[22604,1],1][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> > [yethiraj30][[22604,1],3][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> > [yethiraj30][[22604,1],2][btl_tcp_endpoint.c:636:mca_btl_tcp_endpoint_complete_connect] > connect() to 192.168.0.31 failed: No route to host (113) > >> NNODES=16, MYRANK=10, HOSTNAME=yethiraj31 > >> NNODES=16, MYRANK=12, HOSTNAME=yethiraj31 > >> > >> I am not sure how to resolve this issue. In general, I can go from one > machine to another without any problem using ssh. But, when I am trying to > run openmpi over both the machines, I get this error. Any help will be > appreciated. > >> > >> Jagannath > >> ___ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ___ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > ___ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] huge VmRSS on rank 0 after MPI_Init when using "btl_openib_receive_queues" option
hi, does anyone have a clue here ? éloi On 22/04/2011 08:52, Eloi Gaudry wrote: it varies with the receive_queues specification *and* with the number of mpi processes: memory_consumed = nb_mpi_process * nb_buffers * (buffer_size + low_buffer_count_watermark + credit_window_size ) éloi On 04/22/2011 12:26 AM, Jeff Squyres wrote: Does it vary exactly according to your receive_queues specification? On Apr 19, 2011, at 9:03 AM, Eloi Gaudry wrote: hello, i would like to get your input on this: when launching a parallel computation on 128 nodes using openib and the "-mca btl_openib_receive_queues P,65536,256,192,128" option, i observe a rather large resident memory consumption (2GB: 65336*256*128) on the process with rank 0 (and only this process) just after a call to MPI_Init. i'd like to know why the other processes doesn't behave the same: - other processes located on the same nodes don't use that amount of memory - all others processes (i.e. located on any other nodes) neither i'm using OpenMPI-1.4.2, built with gcc-4.3.4 and '--enable-cxx-exceptions --with-pic --with-threads=posix' options. thanks for your help, éloi -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be Direct Phone Number: +32 10 495 147 ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Eloi Gaudry Senior Product Development Engineer Free Field Technologies Company Website: http://www.fft.be Direct Phone Number: +32 10 495 147
[OMPI users] BLCR support not building on 1.5.3
Hello all. I'm building 1.5.3 from source on a Debian Squeeze AMD64 system, and trying to get BLCR support built-in. I've installed all the packages that I think should be relevant to BLCR support, including: +blcr-dkms +libcr0 +libcr-dev +blcr-util I've also installed blcr-testuite . I only run Open MPI's configure after loading the blcr modules, and the tests in blcr-testsuite pass. The relevant headers seem to be in /usr/include and the relevant libraries in /usr/lib . I've tried three different invocations of configure: 1. No BLCR-related arguments. Output snippet from configure: checking --with-blcr value... simple ok (unspecified) checking --with-blcr-libdir value... simple ok (unspecified) checking if MCA component crs:blcr can compile... no 2. With --with-blcr=/usr only Output snippet from configure: checking --with-blcr value... sanity check ok (/usr) checking --with-blcr-libdir value... simple ok (unspecified) configure: WARNING: BLCR support requested but not found. Perhaps you need to specify the location of the BLCR libraries. configure: error: Aborting. 3. With --with-blcr-libdir=/usr/lib only Output snippet from configure: checking --with-blcr value... simple ok (unspecified) checking --with-blcr-libdir value... sanity check ok (/usr/lib) checking if MCA component crs:blcr can compile... no config.log only seems to contain the output of whatever tests were run to determine whether or not blcr support could be compiled, but I don't see any way to get details on what code and compile invocation actually failed, in order to get to the root of the problem. I'm not a configure or m4 expert, so I'm not sure how to go further in troubleshooting this. Help would be much appreciated. Thanks!
Re: [OMPI users] Symbol not found: _evsignal_base
Hi Jeff & others - Thanks for the response. OpenMPI compiles fine, so there are no problems with the make process, but I can't run "mpirun" because the program won't compile with mpicc - this case isn't really covered by the "Getting Help" page. However, I have provided the output of configure/make/make install, even though it was successful. The .tar.gz file is here: http://files.charlesmartinreid.com/openmpi_out.tar.gz If there is any additional info that would be helpful, please let me know. Charles On Wed, May 25, 2011 at 17:27, Jeff Squyres wrote: > This looks like your installation is busted somehow. Can you send all the > information listed here: > >http://www.open-mpi.org/community/help/ > > > On May 24, 2011, at 4:05 PM, charles reid wrote: > > > Hi - > > > > I'm trying to compile a simple hello world program with mpicc, > > > > $ cat test.c > > #include > > > > main() > > { > > printf ("Hello World!\n"); > > } > > > > > > but I'm seeing this issue: > > > > $ ~/pkg/openmpi/1.4.3_bigmac/bin/mpicc test.c > > dyld: Symbol not found: _evsignal_base > > Referenced from: /uufs/ > chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib > > Expected in: flat namespace > > in /uufs/ > chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib > > Trace/BPT trap > > > > > > I found this previous thread, > http://comments.gmane.org/gmane.comp.clustering.open-mpi.user/13033 , > which suggested adding the installation directory's lib/ to LD_LIBRARY_PATH > would fix things, but it did not: > > > > $ export > LD_LIBRARY_PATH="${HOME}/pkg/openmpi/1.4.3_bigmac/lib:${LD_LIBRARY_PATH}"; > ~/pkg/openmpi/1.4.3_bigmac/bin/mpicc test.c > > dyld: Symbol not found: _evsignal_base > > Referenced from: /uufs/ > chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib > > Expected in: flat namespace > > in /uufs/ > chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib > > Trace/BPT trap > > > > > > Any suggestions on what I might be doing wrong? > > > > > > Charles > > ___ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
Re: [OMPI users] Symbol not found: _evsignal_base
You can disregard, the problem has been resolved by re-running the configure/make/make install procedure. Thanks again for the help! Charles On Thu, May 26, 2011 at 13:16, charles reid wrote: > Hi Jeff & others - > > Thanks for the response. OpenMPI compiles fine, so there are no problems > with the make process, but I can't run "mpirun" because the program won't > compile with mpicc - this case isn't really covered by the "Getting Help" > page. However, I have provided the output of configure/make/make install, > even though it was successful. The .tar.gz file is here: > > http://files.charlesmartinreid.com/openmpi_out.tar.gz > > If there is any additional info that would be helpful, please let me know. > > > Charles > > > > On Wed, May 25, 2011 at 17:27, Jeff Squyres wrote: > >> This looks like your installation is busted somehow. Can you send all the >> information listed here: >> >>http://www.open-mpi.org/community/help/ >> >> >> On May 24, 2011, at 4:05 PM, charles reid wrote: >> >> > Hi - >> > >> > I'm trying to compile a simple hello world program with mpicc, >> > >> > $ cat test.c >> > #include >> > >> > main() >> > { >> > printf ("Hello World!\n"); >> > } >> > >> > >> > but I'm seeing this issue: >> > >> > $ ~/pkg/openmpi/1.4.3_bigmac/bin/mpicc test.c >> > dyld: Symbol not found: _evsignal_base >> > Referenced from: /uufs/ >> chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib >> > Expected in: flat namespace >> > in /uufs/ >> chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib >> > Trace/BPT trap >> > >> > >> > I found this previous thread, >> http://comments.gmane.org/gmane.comp.clustering.open-mpi.user/13033 , >> which suggested adding the installation directory's lib/ to LD_LIBRARY_PATH >> would fix things, but it did not: >> > >> > $ export >> LD_LIBRARY_PATH="${HOME}/pkg/openmpi/1.4.3_bigmac/lib:${LD_LIBRARY_PATH}"; >> ~/pkg/openmpi/1.4.3_bigmac/bin/mpicc test.c >> > dyld: Symbol not found: _evsignal_base >> > Referenced from: /uufs/ >> chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib >> > Expected in: flat namespace >> > in /uufs/ >> chpc.utah.edu/common/home/u0552682/pkg/openmpi/1.4.3_bigmac/lib/libopen-pal.0.dylib >> > Trace/BPT trap >> > >> > >> > Any suggestions on what I might be doing wrong? >> > >> > >> > Charles >> > ___ >> > users mailing list >> > us...@open-mpi.org >> > http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > >