My question about version wasn't "why can't you use 1.3?". It was "why do you believe the problems you are seeing are caused by not finding the correct version?".
It looks to me like everything is working correctly, but that communications are blocked for some reason. That doesn't sound like a version mismatch, but rather like there is some reason why the nodes cannot communicate to each other. Note that OMPI isn't complaining about mis-matched messages - it is complaining that it cannot open a socket between various nodes. That has nothing to do with version, and all to do with network permissions/connectivity. Have you tried running this with -mca btl ^tcp ? Perhaps the issue is with the use of TCP for your interconnect. On Apr 26, 2010, at 12:48 PM, Matthew MacManes wrote: > Hi Ralph, > > Its a no-go with the --enable-mpirun-prefix-by-default. > > Version issue: The program I am trying to run (RAY: > http://sourceforge.net/apps/mediawiki/denovoassembler/index.php?title=Main_Page#Installation) > will not work with earlier versions of OpenMPI- this is confirmed both by > the author and by my observations.. > > Any other suggestions? > > > On Mon, Apr 26, 2010 at 08:48, Ralph Castain <r...@open-mpi.org> wrote: > When configuring OMPI. Your configure should look like this: > > ./configure --prefix=<wherever> --enable-mpirun-prefix-by-default ..... > > Just curious: what convinces you that you have a version mismatch? > Connectivity failures can occur for a variety of reasons - this looks more > like you have some kind of network access issue. > > On Apr 26, 2010, at 9:39 AM, Matthew MacManes wrote: > >> Hi Ralph, >> >> Thanks! Do you mean to pass '--enable-mpirun-prefix-by-default' when >> configuring OpenMPI, or when configuring the program I am trying to use. >> Sorry if this should be obvious! >> >> On Mon, Apr 26, 2010 at 08:13, Ralph Castain <r...@open-mpi.org> wrote: >> First, is the directory where you installed OMPI 1.4.1 visible to all the >> nodes? If not, then this won't work. >> >> If it is, then try configuring with --enable-mpirun-prefix-by-default, and >> be sure you specify a prefix that points to your installation. >> >> >> On Apr 26, 2010, at 9:08 AM, Matthew MacManes wrote: >> >>> I am using SGE to submit jobs to one of the TeraGrid sites, specifically >>> TACC-RANGER. The problem, is, that I am using a program that requires >>> OpenMPI version 1.4.1, and the latest install on RANGER is 1.3.1. I was >>> told that I could install OpenMPI in my home directory, and run jobs using >>> my newer version.. However, I am having problems doing this, getting the >>> error message seen below. >>> >>> Its seems that the compute nodes are not accessing all the sufficient >>> libraries for the newer version of OpenMPI. >>> >>> Can anybody tell me what I can do to get the jobs running using the newer >>> version of OpenMPI. Thanks! >>> >>> TACC: Setting memory limits for job 1349843 to 3984588 KB >>> TACC: Dumping job script: >>> ------------------------------ >>> -------------------------------------------------- >>> #!/bin/bash >>> export TMPDIR=$SCRATCH/abyss_tmp/ >>> LD_LIBRARY_PATH=/work/01301/mmacmane >>> LD_LIBRARY_PATH=/work/01301/mmacmane/bin >>> LD_LIBRARY_PATH=/work/01301/mmacmane/include >>> LD_LIBRARY_PATH=/work/01301/mmacmane/etc >>> LD_LIBRARY_PATH=/work/01301/mmacmane/lib >>> LD_LIBRARY_PATH=/work/01301/mmacmane/openmpi-1.4.1 >>> cd /work/01301/mmacmane/Ray-0.0.6 >>> module load openmpi >>> #$ -N testing_MRNA2 >>> #$ -j y >>> #$ -o /work/01301/mmacmane/Ray-0.0.6/testing_MRNA2 >>> #$ -pe 8way 128 >>> #$ -q normal >>> #$ -l h_rt=2:00:00 >>> #$ -M macma...@gmail.com >>> #$ -m be >>> #$ -cwd >>> #$ -V >>> /work/01301/mmacmane/bin/mpirun Ray >>> /work/01301/mmacmane/Ray-0.0.6/Ray_snp.txt-------------------------------------------------------------------------------- >>> TACC: Done. >>> Module mvapich superceded >>> >>> Ray Copyright (C) 2010 Sébastien Boisvert, Jacques Corbeil, François >>> Laviolette >>> http://denovoassembler.sf.net/ >>> This program comes with ABSOLUTELY NO WARRANTY. >>> This is free software, and you are welcome to redistribute it >>> under certain conditions; see "gpl-3.0.txt" for details. >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],114][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],119][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],123][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],17][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],42][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],44][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],1][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],13][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],9][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],104][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],106][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],102][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],45][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],83][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],84][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],92][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],66][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],70][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],52][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],60][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],58][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],72][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],23][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],29][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],31][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],35][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],43][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],0][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],6][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],14][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],73][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],77][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],75][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],99][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],109][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],103][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],51][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],55][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],57][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],113][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],116][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],115][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],19][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],21][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],27][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],37][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],47][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],33][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],8][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],10][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],4][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],97][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],101][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],107][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],82][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],85][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],90][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],79][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],65][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],67][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],61][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],53][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],59][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],127][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],121][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],124][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],18][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],25][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],28][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],39][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],34][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],38][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],3][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],2][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],12][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],105][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],108][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],111][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],91][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],80][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],87][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],69][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],68][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],71][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],63][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],48][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],49][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],16][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],24][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],26][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],125][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],122][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],126][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],40][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],41][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],46][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],15][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],7][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],11][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],100][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],110][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],96][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],88][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],89][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],94][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],76][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],64][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],78][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],62][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],50][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],56][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],20][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],22][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> >>> [i161-311.ranger.tacc.utexas.edu][[29053,1],30][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.5.170 failed: No route to host (113) >>> connect() to 192.168.5.170 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],118][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],112][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],117][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],32][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i128-412.ranger.tacc.utexas.edu][[29053,1],36][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.4.244 failed: No route to host (113) >>> [i120-302.ranger.tacc.utexas.edu][[29053,1],5][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.13.99 failed: No route to host (113) >>> [i156-212.ranger.tacc.utexas.edu][[29053,1],98][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.16.232 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],86][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],95][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],93][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: No route to host (113) >>> [i105-104.ranger.tacc.utexas.edu][[29053,1],74][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.15.0 failed: No route to host (113) >>> [i116-312.ranger.tacc.utexas.edu][[29053,1],54][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.7.64 failed: No route to host (113) >>> [i180-212.ranger.tacc.utexas.edu][[29053,1],120][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.12.104 failed: Connection timed out (110) >>> [i170-204.ranger.tacc.utexas.edu][[29053,1],81][btl_tcp_endpoint.c:638:mca_btl_tcp_endpoint_complete_connect] >>> connect() to 192.168.2.196 failed: Connection timed out (110) >>> -------------------------------------------------------------------------- >>> A daemon (pid 24537) died unexpectedly with status 137 while attempting >>> to launch so we are aborting. >>> >>> There may be more information reported by the environment (see above). >>> >>> This may be because the daemon was unable to find all the needed shared >>> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the >>> location of the shared libraries on the remote nodes and this will >>> automatically be forwarded to the remote nodes. >>> ------------------------------ -------------------------------------------- >>> -------------------------------------------------------------------------- >>> mpirun noticed that the job aborted, but has no info as to the process >>> that caused that situation. >>> -------------------------------------------------------------------------- >>> [i120-302.ranger.tacc.utexas.edu:24530] [[29053,0],0]-[[29053,0],4] >>> mca_oob_tcp_msg_recv: readv failed: Connection reset by peer (104) >>> -------------------------------------------------------------------------- >>> mpirun was unable to cleanly terminate the daemons on the nodes shown >>> below. Additional manual cleanup may be required - please refer to >>> the "orte-clean" tool for assistance. >>> -------------------------------------------------------------------------- >>> i128-412.ranger.tacc.utexas.edu >>> i105-104.ranger.tacc.utexas.edu >>> i170-204.ranger.tacc.utexas.edu >>> [i161-311.ranger.tacc.utexas.edu:28177] [[29053,0],1] routed:binomial: >>> Connection to lifeline [[29053,0],0] lost >>> [i156-212.ranger.tacc.utexas.edu:16331] [[29053,0],6] routed:binomial: >>> Connection to lifeline [[29053,0],0] lost >>> [i180-212.ranger.tacc.utexas.edu:09688] [[29053,0],7] routed:binomial: >>> Connection to lifeline [[29053,0],0] lost >>> TACC: Cleaning up after job: 1349843 >>> TACC: Done. >>> _________________________________ >>> Matthew MacManes >>> PhD Candidate >>> University of California- Berkeley >>> Museum of Vertebrate Zoology >>> Phone: 510-495-5833 >>> Lab Website: http://ib.berkeley.edu/labs/lacey >>> Personal Website: http://macmanes.com/ > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
