Re: [OMPI users] Open MPI:Problem with 64-bit openMPI andintel compiler

[Jeff Squyres]

On Jul 23, 2009, at 11:14 PM, Ralph Castain wrote:


3. get a multi-node allocation and run "pbsdsh echo $LD_LIBRARY_PATH"
and see what libs you are defaulting to on the other nodes.




Be careful with this one; you want to ensure that your local shell  
doesn't expand $LD_LIBRARY_PATH and simply display the same value on  
all nodes.  It might be easiest to write a 2 line script and run that:


$ cat myscript
#!/bin/sh
echo LD_LIB_PATH on `hostname` is: $LD_LIBRARY_PATH
$ chmod +x myscript
$ pdsh myscript

--
Jeff Squyres
jsquy...@cisco.com

Re: [OMPI users] Open MPI:Problem with 64-bit openMPI andintel compiler

[Ralph Castain]

Good point.

Other thing you might consider (though it is unlikely to be part of  
this problem) is upgrading to 1.3.3. It probably isn't a good idea to  
be using a release candidate for anything serious.



On Jul 24, 2009, at 5:21 AM, Jeff Squyres wrote:


On Jul 23, 2009, at 11:14 PM, Ralph Castain wrote:


3. get a multi-node allocation and run "pbsdsh echo $LD_LIBRARY_PATH"
and see what libs you are defaulting to on the other nodes.




Be careful with this one; you want to ensure that your local shell  
doesn't expand $LD_LIBRARY_PATH and simply display the same value on  
all nodes.  It might be easiest to write a 2 line script and run that:


$ cat myscript
#!/bin/sh
echo LD_LIB_PATH on `hostname` is: $LD_LIBRARY_PATH
$ chmod +x myscript
$ pdsh myscript

--
Jeff Squyres
jsquy...@cisco.com

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Open MPI:Problem with 64-bit openMPIandintel compiler

[jimkress_58]

You can avoid the "library confusion problem" by building 64 bit and 32 bit
version of openMPI in two different directories and then use mpi-selector
(on your head and compute nodes) to switch between the two.

Just my $0.02

Jim

-Original Message-
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On
Behalf Of Jeff Squyres
Sent: Friday, July 24, 2009 7:22 AM
To: Open MPI Users
Subject: Re: [OMPI users] Open MPI:Problem with 64-bit openMPIandintel
compiler

On Jul 23, 2009, at 11:14 PM, Ralph Castain wrote:

> 3. get a multi-node allocation and run "pbsdsh echo $LD_LIBRARY_PATH"
> and see what libs you are defaulting to on the other nodes.
>


Be careful with this one; you want to ensure that your local shell  
doesn't expand $LD_LIBRARY_PATH and simply display the same value on  
all nodes.  It might be easiest to write a 2 line script and run that:

$ cat myscript
#!/bin/sh
echo LD_LIB_PATH on `hostname` is: $LD_LIBRARY_PATH
$ chmod +x myscript
$ pdsh myscript

-- 
Jeff Squyres
jsquy...@cisco.com

___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] Missmatch between sent and recieved data

[Alexey Sokolov]

Hi

I'm sorry if the answer to my question is very simple, but I'm really
confused with this situation.

I got a program that now contains 1 master and 1 slave processes. Master
process sends few similar chars to the slave process with this:

for (unsigned i = 0; i < SomeVariable; i++) {
  strcpy(chdata[i], "asdf");
  MPI_Isend(chdata[i], 4, MPI_CHAR, SLAVE, 1, MPI_COMM_WORLD,
&req[i]);
}

And the slave process receives them with this:

for (unsigned i = 0; i < SomeVariable; i++) {
  char * buf = new char[10];
  MPI_Recv(buf, 4, MPI_CHAR, MASTER, 1, MPI_COMM_WORLD, &stat);
  printf("Received data [%s] of length %d\n", buf, strlen(buf));
}

As I understand output should be "Received data [asdf] of length 4", but
instead of this I got my 'asdf' with additional 3 random ANSI characters
in string of length 7.

I've already spent few hours to find the mistake out but unsuccessfully.
I do successfully transfer int data in the similar way, but this case
make me crazy. Could you help me to find out where did I fail in this?

Sincerely yours, Alexey. 

Re: [OMPI users] Missmatch between sent and recieved data

[Dorian Krause]

Hi,

you do not send the trailing '0' which is used to determine the 
stringlength. I assume that chdata[i] has at least length 5 (otherwise 
you overrun your memory). Replace the "4" by "5" in MPI_Isend and 
MPI_Recv and everything should work (If I get the problem right).


Dorian.


Alexey Sokolov wrote:

Hi

I'm sorry if the answer to my question is very simple, but I'm really
confused with this situation.

I got a program that now contains 1 master and 1 slave processes. Master
process sends few similar chars to the slave process with this:

for (unsigned i = 0; i < SomeVariable; i++) {
  strcpy(chdata[i], "asdf");
  MPI_Isend(chdata[i], 4, MPI_CHAR, SLAVE, 1, MPI_COMM_WORLD,
&req[i]);
}

And the slave process receives them with this:

for (unsigned i = 0; i < SomeVariable; i++) {
  char * buf = new char[10];
  MPI_Recv(buf, 4, MPI_CHAR, MASTER, 1, MPI_COMM_WORLD, &stat);
  printf("Received data [%s] of length %d\n", buf, strlen(buf));
}

As I understand output should be "Received data [asdf] of length 4", but
instead of this I got my 'asdf' with additional 3 random ANSI characters
in string of length 7.

I've already spent few hours to find the mistake out but unsuccessfully.
I do successfully transfer int data in the similar way, but this case
make me crazy. Could you help me to find out where did I fail in this?

Sincerely yours, Alexey. 


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

  

Re: [OMPI users] Receiving an unknown number of messages

[Shaun Jackman]

Eugene Loh wrote:

Shaun Jackman wrote:

Eugene Loh wrote:

Shaun Jackman wrote:

For my MPI application, each process reads a file and for each line 
sends a message (MPI_Send) to one of the other processes determined 
by the contents of that line. Each process posts a single MPI_Irecv 
and uses MPI_Request_get_status to test for a received message. If a 
message has been received, it processes the message and posts a new 
MPI_Irecv. I believe this situation is not safe and prone to 
deadlock since MPI_Send may block. The receiver would need to post 
as many MPI_Irecv as messages it expects to receive, but it does not 
know in advance how many messages to expect from the other 
processes. How is this situation usually handled in an MPI 
appliation where the number of messages to receive is unknown?

...


Each process posts an MPI_Irecv to listen for in-coming messages.

Each process enters a loop in which it reads its file and sends out 
messages.  Within this loop, you also loop on MPI_Test to see if any 
message has arrived.  If so, process it, post another MPI_Irecv(), 
and keep polling.  (I'd use MPI_Test rather than 
MPI_Request_get_status since you'll have to call something like 
MPI_Test anyhow to complete the receive.)


Once you've posted all your sends, send out a special message to 
indicate you're finished.  I'm thinking of some sort of tree 
fan-in/fan-out barrier so that everyone will know when everyone is 
finished.


Keep polling on MPI_Test, processing further receives or advancing 
your fan-in/fan-out barrier.


So, the key ingredients are:

*) keep polling on MPI_Test and reposting MPI_Irecv calls to drain 
in-coming messages while you're still in your "send" phase
*) have another mechanism for processes to notify one another when 
they've finished their send phases

Hi Eugene,

Very astute. You've pretty much exactly described how it works now, 
particularly the loop around MPI_Test and MPI_Irecv to drain incoming 
messages. So, here's my worry, which I'll demonstrate with an example. 
We have four processes. Each calls MPI_Irecv once. Each reads one line 
of its file. Each sends one message with MPI_Send to some other 
process based on the line that it has read, and then goes into the 
MPI_Test/MPI_Irecv loop.


The events fall out in this order
2 sends to 0 and does not block (0 has one MPI_Irecv posted)
3 sends to 1 and does not block (1 has one MPI_Irecv posted)
0 receives the message from 2, consuming its MPI_Irecv
1 receives the message from 3, consuming its MPI_Irecv
0 sends to 1 and blocks (1 has no more MPI_Irecv posted)
1 sends to 0 and blocks (0 has no more MPI_Irecv posted)
and now processes 0 and 1 are deadlocked.

When I say `receives' above, I mean that Open MPI has received the 
message and copied it into the buffer passed to the MPI_Irecv call, 
but the application hasn't yet called MPI_Test. The next step would be 
for all the processes to call MPI_Test, but 0 and 1 are already 
deadlocked.


I don't get it.  Processes should drain aggressively.  So, if 0 receives 
a message, it should immediately post the next MPI_Irecv.  Before 0 
posts a send, it should MPI_Test (and post the next MPI_Irecv if the 
test received a message).


Further, you could convert to MPI_Isend.

But maybe I'm missing something.


Hi Eugene,

Before posting a send, the process can call MPI_Test to check for a 
received packet, but there's a race condition here. The packet can 
arrive between the MPI_Test (which returns false) and before it calls 
MPI_Send. I've added the MPI_Test calls to my example scenario:


2 calls MPI_Test. No message is waiting, so 2 decides to send.
2 sends to 0 and does not block (0 has one MPI_Irecv posted)
3 calls MPI_Test. No message is waiting, so 3 decides to send.
3 sends to 1 and does not block (1 has one MPI_Irecv posted)
0 calls MPI_Test. No message is waiting, so 0 decides to send.
0 receives the message from 2, consuming its MPI_Irecv
1 calls MPI_Test. No message is waiting, so 1 decides to send.
1 receives the message from 3, consuming its MPI_Irecv
0 sends to 1 and blocks (1 has no more MPI_Irecv posted)
1 sends to 0 and blocks (0 has no more MPI_Irecv posted)
and now processes 0 and 1 are deadlocked.

Cheers,
Shaun

Re: [OMPI users] Receiving an unknown number of messages

[Eugene Loh]

Shaun Jackman wrote:


2 calls MPI_Test. No message is waiting, so 2 decides to send.
2 sends to 0 and does not block (0 has one MPI_Irecv posted)
3 calls MPI_Test. No message is waiting, so 3 decides to send.
3 sends to 1 and does not block (1 has one MPI_Irecv posted)
0 calls MPI_Test. No message is waiting, so 0 decides to send.
0 receives the message from 2, consuming its MPI_Irecv
1 calls MPI_Test. No message is waiting, so 1 decides to send.
1 receives the message from 3, consuming its MPI_Irecv
0 sends to 1 and blocks (1 has no more MPI_Irecv posted)
1 sends to 0 and blocks (0 has no more MPI_Irecv posted)
and now processes 0 and 1 are deadlocked.


I'm in over my head here, but let me try.

Okay, so the problem is 0 sends to 1 and 1 sends to 0 so they both 
lock.  The usual way around this is for each process first to post an 
MPI_Irecv, but that might be consumed by some "third party" sends.  So, 
you lock.


Still, why can't you use non-blocking sends?  Use MPI_Isend.  As you 
wait for its completion, you can process in-coming messages.

Re: [OMPI users] Receiving an unknown number of messages

[Shaun Jackman]

Hi Prasadcse,

At each iteration, each process chooses one other process to send to 
(based on the line it has just read from a file). If we send the 
number of packets to expect in advance, we then need to send an 
`expect zero packets from me' message to all the processes that we 
don't have a message for, because this method depends on every process 
knowing exactly how many messages to expect from every other process 
at each iteration.


Cheers,
Shaun

Prasadcse Perera wrote:
I think its a real good way to use MPI_Irecv/MPI_Test on the receiver 
side to avoid any blocks which sender might run in to. But I'm a bit 
curious on the fact, Can't we use a special message beforehand between 
the sender/receivers to let the receivers know how many messages to 
expect ? This way the sender and the receiver can safely loop through 
without blocking I believe. Also even with the MPI_Irecv/MPI_Test it 
will serve as an extra proof for the receivers to proceed. Any ideas on 
that ?


On Wed, Jul 15, 2009 at 2:15 AM, Eugene Loh > wrote:


Shaun Jackman wrote:

For my MPI application, each process reads a file and for each
line sends a message (MPI_Send) to one of the other processes
determined by the contents of that line. Each process posts a
single MPI_Irecv and uses MPI_Request_get_status to test for a
received message. If a message has been received, it processes
the message and posts a new MPI_Irecv. I believe this situation
is not safe and prone to deadlock since MPI_Send may block. The
receiver would need to post as many MPI_Irecv as messages it
expects to receive, but it does not know in advance how many
messages to expect from the other processes. How is this
situation usually handled in an MPI appliation where the number
of messages to receive is unknown?

In a non-MPI network program I would create one thread for
receiving and processing, and one thread for transmitting. Is
threading a good solution? Is there a simpler solution?

Under what conditions will MPI_Send block and under what
conditions will it definitely not block?


I haven't seen any other responses, so I'll try.

The conditions under which MPI_Send will block are implementation
dependent.  Even for a particular implementation, the conditions may
be tricky to describe -- e.g., what interconnect is being used to
reach the peer, is there any congestion, etc.  I guess you could use
buffered sends... or maybe you can't if you really don't know how
many messages will be sent.

Let's just assume they'll block.

I'm unsure what overall design you're suggesting, so I'll suggest
something.

Each process posts an MPI_Irecv to listen for in-coming messages.

Each process enters a loop in which it reads its file and sends out
messages.  Within this loop, you also loop on MPI_Test to see if any
message has arrived.  If so, process it, post another MPI_Irecv(),
and keep polling.  (I'd use MPI_Test rather than
MPI_Request_get_status since you'll have to call something like
MPI_Test anyhow to complete the receive.)

Once you've posted all your sends, send out a special message to
indicate you're finished.  I'm thinking of some sort of tree
fan-in/fan-out barrier so that everyone will know when everyone is
finished.

Keep polling on MPI_Test, processing further receives or advancing
your fan-in/fan-out barrier.

So, the key ingredients are:

*) keep polling on MPI_Test and reposting MPI_Irecv calls to drain
in-coming messages while you're still in your "send" phase
*) have another mechanism for processes to notify one another when
they've finished their send phases

Re: [OMPI users] Receiving an unknown number of messages

[Shaun Jackman]

Eugene Loh wrote:

Shaun Jackman wrote:


2 calls MPI_Test. No message is waiting, so 2 decides to send.
2 sends to 0 and does not block (0 has one MPI_Irecv posted)
3 calls MPI_Test. No message is waiting, so 3 decides to send.
3 sends to 1 and does not block (1 has one MPI_Irecv posted)
0 calls MPI_Test. No message is waiting, so 0 decides to send.
0 receives the message from 2, consuming its MPI_Irecv
1 calls MPI_Test. No message is waiting, so 1 decides to send.
1 receives the message from 3, consuming its MPI_Irecv
0 sends to 1 and blocks (1 has no more MPI_Irecv posted)
1 sends to 0 and blocks (0 has no more MPI_Irecv posted)
and now processes 0 and 1 are deadlocked.


I'm in over my head here, but let me try.

Okay, so the problem is 0 sends to 1 and 1 sends to 0 so they both 
lock.  The usual way around this is for each process first to post an 
MPI_Irecv, but that might be consumed by some "third party" sends.  So, 
you lock.


Still, why can't you use non-blocking sends?  Use MPI_Isend.  As you 
wait for its completion, you can process in-coming messages.


Hi Eugene,

Using MPI_Isend may be the ultimate solution. If it wasn't necessary, 
I was hoping to avoid it. MPI_Isend has the added complexity of 
needing to store all the outgoing messages until the send completes. 
MPI_Ibsend allows MPI to handle that complexity for us, but requires 
preallocating a buffer for MPI_Buffer_attach.


If there were some upper bound on the number of messages that we can 
expect to receive per iteration (say one per process), an alternative 
might be to post more than one MPI_Irecv. Or perhaps, posting one 
MPI_Irecv per process setting the `source' parameter would prevent the 
"third-party" situation.


Cheers,
Shaun