Re: [OMPI users] Debugging memory use of Open MPI

2009-04-16 Thread Eugene Loh
Shaun Jackman wrote: Eugene Loh wrote: What's the rest? I said the shared-memory area is much smaller, but I was confused about which OMPI release I was using. So, the shared-memory area was 128 Mbyte and it was getting mapped in once for each process, and so it was counted doubly. If th

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-16 Thread Shaun Jackman
Eugene Loh wrote: ... What's the rest? I said the shared-memory area is much smaller, but I was confused about which OMPI release I was using. So, the shared-memory area was 128 Mbyte and it was getting mapped in once for each process, and so it was counted doubly. If there are eight proces

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-16 Thread Eugene Loh
Eugene Loh wrote: Shaun Jackman wrote: What's the purpose of the 400 MB that MPI_Init has allocated? It's for... um, I don't know. Let's see... About a third of it appears to be vt_open() -> VTThrd_open() -> VTGen_open which I'm guessing is due to the VampirTrace instrumentation (maybe al

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Chris Gottbrath
Eugene, On Apr 14, 2009, at 7:10 PM, Eugene Loh wrote: > Shaun Jackman wrote: > >> Wow. Thanks, Eugene. I definitely have to look into the Sun HPC >> ClusterTools. It looks as though it could be very informative. > > Great. And, I didn't mean to slight TotalView. I'm just not > familiar wit

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Eugene Loh
Shaun Jackman wrote: Wow. Thanks, Eugene. I definitely have to look into the Sun HPC ClusterTools. It looks as though it could be very informative. Great. And, I didn't mean to slight TotalView. I'm just not familiar with it. What's the purpose of the 400 MB that MPI_Init has allocated?

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Chris Gottbrath
Shaun, These all look like fine suggestions. Another tool you should consider using for this problem or others like it in the future is TotalView. It seems like there are two related questions in your current troubleshooting scenario: 1. is the memory being used where you think it is? 2. is t

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Shaun Jackman
Eugene Loh wrote: Okay. Attached is a "little" note I wrote up illustrating memory profiling with Sun tools. (It's "big" because I ended up including a few screenshots.) The program has a bunch of one-way message traffic and some user-code memory allocation. I then rerun with the receiver

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Eugene Loh
Shaun Jackman wrote: Eugene Loh wrote: On the other hand, I assume the memory imbalance we're talking about is rather severe. Much more than 2500 bytes to be noticeable, I would think. Is that really the situation you're imagining? The memory imbalance is drastic. I'm expecting 2 GB of m

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Shaun Jackman
Eugene Loh wrote: ompi_info -a | grep eager depends on the BTL. E.g., sm=4K but tcp is 64K. self is 128K. Thanks, Eugene. On the other hand, I assume the memory imbalance we're talking about is rather severe. Much more than 2500 bytes to be noticeable, I would think. Is that really the s

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Eugene Loh
On Apr 14, 2009, at 12:02 PM, Shaun Jackman wrote: Assuming the problem is congestion and that messages are backing up, ... I'd check this assumption first before going too far down that path. You might be able to instrument your code to spit out sends and receives. VampirTrace (and PERU

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Ralph Castain
On Apr 14, 2009, at 12:02 PM, Shaun Jackman wrote: Hi Eugene, Eugene Loh wrote: At 2500 bytes, all messages will presumably be sent "eagerly" -- without waiting for the receiver to indicate that it's ready to receive that particular message. This would suggest congestion, if any, is on

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Eugene Loh
Shaun Jackman wrote: Eugene Loh wrote: At 2500 bytes, all messages will presumably be sent "eagerly" -- without waiting for the receiver to indicate that it's ready to receive that particular message. This would suggest congestion, if any, is on the receiver side. Some kind of congestion c

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-14 Thread Shaun Jackman
Hi Eugene, Eugene Loh wrote: At 2500 bytes, all messages will presumably be sent "eagerly" -- without waiting for the receiver to indicate that it's ready to receive that particular message. This would suggest congestion, if any, is on the receiver side. Some kind of congestion could, I supp

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-09 Thread Eugene Loh
Shaun Jackman wrote: Eugene Loh wrote: I'm no expert, but I think it's something like this: 1) If the messages are short, they're sent over to the receiver. If the receiver does not expect them (no MPI_Irecv posted), it buffers them up. 2) If the messages are long, only a little bit is s

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-09 Thread Shaun Jackman
Eugene Loh wrote: I'm no expert, but I think it's something like this: 1) If the messages are short, they're sent over to the receiver. If the receiver does not expect them (no MPI_Irecv posted), it buffers them up. 2) If the messages are long, only a little bit is sent over to the receiver

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-09 Thread Shaun Jackman
Eugene Loh wrote: I'm no expert, but I think it's something like this: 1) If the messages are short, they're sent over to the receiver. If the receiver does not expect them (no MPI_Irecv posted), it buffers them up. 2) If the messages are long, only a little bit is sent over to the receiver

Re: [OMPI users] Debugging memory use of Open MPI

2009-04-09 Thread Eugene Loh
Shaun Jackman wrote: When running my Open MPI application, I'm seeing three processors that are using five times as much memory as the others when they should all use the same amount of memory. To start the debugging process, I would like to know if it's my application or the Open MPI library

[OMPI users] Debugging memory use of Open MPI

2009-04-09 Thread Shaun Jackman
When running my Open MPI application, I'm seeing three processors that are using five times as much memory as the others when they should all use the same amount of memory. To start the debugging process, I would like to know if it's my application or the Open MPI library that's using the addit