I have a small test code that I've managed to duplicate the results from a larger code. In essence, using the sm btl with ISend, I wind up with the communication being completely serialized, i.e., all the calls from process 1 complete, then all from 2, ...
This is version 1.3.2, vanilla compile. I get the same results on my RHEL5 nehalem and an OS X laptop. Here's an example of the output (note: there is a usleep in the code to mimick my computation loop, and ensure that this is not a simple I/O sequencing issue): ---- Ignore the "next" in the output below, it was a broadcast test. mpirun -np 5 ./mpi_split_test Master [id = 0 of 5] is running on bolstadm-lm1 [0] next = 10 Server [id = 3, 2, 1 of 5] is running on bolstadm-lm1 Compositor [id = 1, 0 of 5] is running on bolstadm-lm1 [1] next = 10 Sending buffer 0 from 1 Server [id = 2, 1, 0 of 5] is running on bolstadm-lm1 [2] next = 10 Sending buffer 0 from 2 [3] next = 10 Server [id = 4, 3, 2 of 5] is running on bolstadm-lm1 [4] next = 10 Sending buffer 0 from 3 Sending buffer 1 from 1 Sending buffer 1 from 2 Sending buffer 1 from 3 Sending buffer 2 from 1 Sending buffer 2 from 2 Sending buffer 2 from 3 Sending buffer 3 from 1 Sending buffer 3 from 2 Sending buffer 4 from 1 Receiving buffer from 1, buffer = hello from 1 for the 0 time Receiving buffer from 1, buffer = hello from 1 for the 1 time Sending buffer 4 from 2 Sending buffer 4 from 3 Sending buffer 5 from 1 Receiving buffer from 1, buffer = hello from 1 for the 2 time Sending buffer 6 from 1 Receiving buffer from 1, buffer = hello from 1 for the 3 time -----At this point, processes 2 & 3 are stuck in an MPI_Wait ... Sending buffer 9 from 1 Receiving buffer from 1, buffer = hello from 1 for the 6 time Receiving buffer from 1, buffer = hello from 1 for the 7 time Receiving buffer from 1, buffer = hello from 1 for the 8 time Receiving buffer from 1, buffer = hello from 1 for the 9 time Receiving buffer from 2, buffer = hello from 2 for the 0 time Receiving buffer from 2, buffer = hello from 2 for the 1 time Receiving buffer from 2, buffer = hello from 2 for the 2 time Sending buffer 5 from 2 Sending buffer 6 from 2 Receiving buffer from 2, buffer = hello from 2 for the 3 time ---- Now process 2 is now running, 1 is in a barrier, 3 is still in Wait .... Sending buffer 9 from 2 Receiving buffer from 2, buffer = hello from 2 for the 6 time Receiving buffer from 2, buffer = hello from 2 for the 7 time Receiving buffer from 2, buffer = hello from 2 for the 8 time Receiving buffer from 2, buffer = hello from 2 for the 9 time Receiving buffer from 3, buffer = hello from 3 for the 0 time Sending buffer 5 from 3 Receiving buffer from 3, buffer = hello from 3 for the 1 time Receiving buffer from 3, buffer = hello from 3 for the 2 time ---- And now process 3 goes ... Receiving buffer from 3, buffer = hello from 3 for the 8 time Receiving buffer from 3, buffer = hello from 3 for the 9 time Now running under TCP: mpirun --mca btl tcp,self -np 5 ./mpi_split_test Compositor [id = 1, 0 of 5] is running on bolstadm-lm1 Master [id = 0 of 5] is running on bolstadm-lm1 [0] next = 10 Server [id = 2, 1, 0 of 5] is running on bolstadm-lm1 Server [id = 3, 2, 1 of 5] is running on bolstadm-lm1 Server [id = 4, 3, 2 of 5] is running on bolstadm-lm1 [4] next = 10 Sending buffer 0 from 3 Sending buffer 0 from 1 [2] next = 10 [1] next = 10 Sending buffer 0 from 2 [3] next = 10 Receiving buffer from 1, buffer = hello from 1 for the 0 time Receiving buffer from 3, buffer = hello from 3 for the 0 time Receiving buffer from 2, buffer = hello from 2 for the 0 time Sending buffer 1 from 3 Sending buffer 1 from 1 Sending buffer 1 from 2 Receiving buffer from 1, buffer = hello from 1 for the 1 time Receiving buffer from 2, buffer = hello from 2 for the 1 time Receiving buffer from 3, buffer = hello from 3 for the 1 time Sending buffer 2 from 3 Sending buffer 2 from 2 Sending buffer 2 from 1 Receiving buffer from 1, buffer = hello from 1 for the 2 time Receiving buffer from 2, buffer = hello from 2 for the 2 time Receiving buffer from 3, buffer = hello from 3 for the 2 time ... So, has this been reported before? I've seen some messages on the developer list about hanging with the sm btl. I'll post the test code if requested (this email is already long) Mark
