This error indicates the IP address exported by the peer is not reachable. You can use the tcp btl parameters: -mca btl_tcp_include eth0,eth1 or -mca btl_tcp_exclude eth1 To specify the set of interfaces to use/not use.
George was correct - these should be btl_tcp_if_include/btl_tcp_if_exclude. Regards, Tim