I'm trying to run openmpi between two 11.04 virtual machines, each on each own physical node. Each VM has three network interfaces, one is the base (eth0) which is in the same subnet as the hypervisor bridge and the other two are Intel SR-IOV VFs. I can ping across all the interfaces. Bottom line is that I'm trying to run the OSU benchmarks between the two VMs and I get the following error. It's also shown that I can run the two processes on the same VM and it works fine.
hoot@u1-1104:~$ mpirun -host 10.10.10.1,10.10.10.2 osu_bw [u2-1104:1946] *** An error occurred in MPI_Waitall [u2-1104:1946] *** on communicator MPI_COMM_WORLD [u2-1104:1946] *** MPI_ERR_TRUNCATE: message truncated [u2-1104:1946] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort) # OSU MPI Bandwidth Test v3.3 # Size Bandwidth (MB/s) -------------------------------------------------------------------------- mpirun has exited due to process rank 1 with PID 1946 on node 10.10.10.2 exiting without calling "finalize". This may have caused other processes in the application to be terminated by signals sent by mpirun (as reported here). -------------------------------------------------------------------------- hoot@u1-1104:~$ mpirun -host 10.10.10.1,10.10.10.1 osu_bw # OSU MPI Bandwidth Test v3.3 # Size Bandwidth (MB/s) 1 2.87 2 5.88 4 11.21 8 22.53 16 46.84 32 91.84 64 176.93 128 278.83 256 537.92 512 888.02 1024 1602.69 2048 2757.05 4096 2510.99 8192 3504.59 16384 4487.80 32768 4097.11 65536 4100.36 131072 4058.36 262144 4090.21 524288 7335.43 1048576 7523.41 2097152 7165.27 4194304 7548.46 hoot@u1-1104:~$ mpirun -host 10.10.10.2,10.10.10.2 osu_bw # OSU MPI Bandwidth Test v3.3 # Size Bandwidth (MB/s) 1 4.54 2 9.20 4 18.70 8 37.40 16 74.68 32 144.03 64 262.93 128 523.46 256 977.52 512 1732.71 1024 2981.65 2048 4853.07 4096 5493.16 8192 7357.55 16384 9300.16 32768 4879.94 65536 4596.26 131072 4471.06 262144 4559.58 524288 4501.23 1048576 4541.63 2097152 4504.08 4194304 4493.76 hoot@u1-1104:~$ mpirun -host 10.10.10.2,10.10.10.2 osu_bw # OSU MPI Bandwidth Test v3.3 # Size Bandwidth (MB/s) 1 4.50 2 9.14 4 18.51 8 36.47 16 74.05 32 142.71 64 256.99 128 516.84 256 972.40 512 1709.23 1024 2937.36 2048 4903.72 4096 5550.57 8192 7297.00 16384 8908.34 32768 8640.99 65536 8424.97 131072 8059.00 262144 4541.50 524288 4560.11 1048576 4554.80 2097152 4527.91 4194304 4493.71
