Nevermind, found the answer in:
http://www.open-mpi.org/community/lists/users/2005/10/0250.php
"-mca btl_tcp_exclude eth1 "
-Galen
+
Galen Arnold, consulting group--system engineer
National Center for Supercomputing Applications
1205 W. Clark St.(217) 244-3
What was the fix last time [openmpi 1.2 is in action below] ?
[arnoldg@honest3 mpi]$ !mpirun
mpirun --mca btl self,sm,tcp -np 4 -machinefile hosts allall_openmpi_icc
50 50 1000
[honest1][0,1,0][btl_tcp_endpoint.c:572:mca_btl_tcp_endpoint_complete_connect]
connect() failed with errno=113
mpiru
