Re: [OMPI users] printf and scanf problem of C code compiled with Open MPI

[Pascal Deveze]

Maybe this could solve your problem: Just add \n in the string you want 
to display:

printf("Please give N= \n");

Of course, this will return, but the string is displayed. This run by me 
without the fflush().


On the other hand, do you really observe that the time of the scanf () 
and the time to enter "N" be insignificant ?


Pascal

Meilin Bai a écrit :
So it means that MPI doesn't suit to interactive programming? Though 
we can really use fflush(stdout) to get the right order, it takes too 
more time, and it's said that using fflush() is not a good progrmming 
style in C.
 
On the other hand, in Fortran language, this situation won't exist. 
Maybe it is because I/O implement is a built-in part of Fortran, while 
in C/C++ it is realized only through function like scanf, printf, et al?



 
On Wed, Mar 30, 2011 at 2:38 AM, Prentice Bisbal > wrote:


On 03/29/2011 01:29 PM, Meilin Bai wrote:
> Dear open-mpi users:
>
> I come across a little problem when running a MPI C program compiled
> with Open MPI 1.4.3. A part of codes as follows:
>
> MPI_Init(&argc, &argv);
> MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
> MPI_Comm_rank(MPI_COMM_WORLD, &myid);
> MPI_Get_processor_name(processor_name, &namelen);
> if (myid == 0) {
>  printf("Please give N= ");
>  //fflush(stdout);
>  scanf("%d", &n);
>  startwtime = MPI_Wtime();
>  }
>
> If comment out the sentence of "fflush(stdout);", it doesn't
print out
> the message till I input an integer n. And if I add the fflush
function
> between them, it works as expected, though comsumming time
obviously.
>
> However, when I compiled it with Mpich2-1.3.2p1, without fflush
function
> in the code, it works correctly.
>
> Can anyone know what the matter is.
>

The Open MPI Developers (Jeff, Ralph, etc) can confirm this:

The MPI standard doesn't have a lot of strict requirements for I/O
behavior like this, so implementations are allowed to buffer I/O
if they
want. There is nothing wrong with requiring fflush(stdout) in order to
get the behavior you want. In fact, if you check some text books
on MPI
programming, I'm pretty sure they recommend using fflush to minimize
this problem.

MPICH behaves differently because its developers made different design
choices.

Neither behavior is "wrong".

--
Prentice
___
users mailing list
us...@open-mpi.org 
http://www.open-mpi.org/mailman/listinfo.cgi/users




--
Meilin Bai



___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] printf and scanf problem of C code compiled with Open MPI

[Meilin Bai]

You remind me. I now realize that it is not an matter of compiler, but an
issue of C language. The printf() function in C doesn't print messages onto
the standard ouput immediately, but instead stores them in a buffer. Only in
some cases does the standard output work, defined in standard C:
1. The buffer is full or freshed compulsively (like use fflush()).
2. When a newline comes (\n).
3. Needs reading from buffer (such as scanf()).

Some compilers may deal with this function, and compiled by Mpich2 this code
seems be well-off.

Thanks very much.


On Wed, Mar 30, 2011 at 3:39 PM, Pascal Deveze wrote:

>  Maybe this could solve your problem: Just add \n in the string you want to
> display:
> printf("Please give N= \n");
>
> Of course, this will return, but the string is displayed. This run by me
> without the fflush().
>
> On the other hand, do you really observe that the time of the scanf () and
> the time to enter "N" be insignificant ?
>
> Pascal
>
> Meilin Bai a écrit :
>
> So it means that MPI doesn't suit to interactive programming? Though we can
> really use fflush(stdout) to get the right order, it takes too more time,
> and it's said that using fflush() is not a good progrmming style in C.
>
> On the other hand, in Fortran language, this situation won't exist. Maybe
> it is because I/O implement is a built-in part of Fortran, while in C/C++ it
> is realized only through function like scanf, printf, et al?
>
>
>
> On Wed, Mar 30, 2011 at 2:38 AM, Prentice Bisbal  wrote:
>
>>  On 03/29/2011 01:29 PM, Meilin Bai wrote:
>> > Dear open-mpi users:
>> >
>> > I come across a little problem when running a MPI C program compiled
>> > with Open MPI 1.4.3. A part of codes as follows:
>> >
>> > MPI_Init(&argc, &argv);
>> > MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>> > MPI_Comm_rank(MPI_COMM_WORLD, &myid);
>> > MPI_Get_processor_name(processor_name, &namelen);
>> > if (myid == 0) {
>> >  printf("Please give N= ");
>> >  //fflush(stdout);
>> >  scanf("%d", &n);
>> >  startwtime = MPI_Wtime();
>> >  }
>> >
>> > If comment out the sentence of "fflush(stdout);", it doesn't print out
>> > the message till I input an integer n. And if I add the fflush function
>> > between them, it works as expected, though comsumming time obviously.
>> >
>> > However, when I compiled it with Mpich2-1.3.2p1, without fflush function
>> > in the code, it works correctly.
>> >
>> > Can anyone know what the matter is.
>> >
>>
>>  The Open MPI Developers (Jeff, Ralph, etc) can confirm this:
>>
>> The MPI standard doesn't have a lot of strict requirements for I/O
>> behavior like this, so implementations are allowed to buffer I/O if they
>> want. There is nothing wrong with requiring fflush(stdout) in order to
>> get the behavior you want. In fact, if you check some text books on MPI
>> programming, I'm pretty sure they recommend using fflush to minimize
>> this problem.
>>
>> MPICH behaves differently because its developers made different design
>> choices.
>>
>> Neither behavior is "wrong".
>>
>> --
>> Prentice
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>
> --
> Meilin Bai
>
> --
>
> ___
> users mailing 
> listusers@open-mpi.orghttp://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
Meilin Bai
School of Electronic Engineering and Computer Science, Peking University
Beijing 100871, China
E-Mail: meilin@gmail.com
   mei...@pku.edu.cn
MSN: meilin@hotmail.com
Cellular: +86-1342-619-8430

Re: [OMPI users] printf and scanf problem of C code compiled with Open MPI

[Jeff Squyres]

I don't remember this being a function of the C language or the compiler; I'm 
pretty sure you can change whether stdout is line buffered or not at the OS 
level.

As for OMPI, we deliberately set each MPI process' stdout to be line buffered 
before relaying it up to the mpirun process.  This was because we wanted to 
limit non-MPI network traffic (i.e., bundle more output into fewer, longer 
messages, rather than more, shorter messages -- i.e., less overall network and 
CPU overhead).  I don't know if that makes anything more or less friendly to 
interactive computing; I don't think using fflush() or \n is onerous.  

Indeed, stdout / stdin processing likely isn't part of the high performance 
section of your application, so we figured it was an easy tradeoff to make 
(i.e., less network overhead leading to potentially better MPI message 
performance).


On Mar 30, 2011, at 3:31 AM, Meilin Bai wrote:

> You remind me. I now realize that it is not an matter of compiler, but an 
> issue of C language. The printf() function in C doesn't print messages onto 
> the standard ouput immediately, but instead stores them in a buffer. Only in 
> some cases does the standard output work, defined in standard C:
> 1. The buffer is full or freshed compulsively (like use fflush()).
> 2. When a newline comes (\n).
> 3. Needs reading from buffer (such as scanf()).
>  
> Some compilers may deal with this function, and compiled by Mpich2 this code 
> seems be well-off.
>  
> Thanks very much.
> 
>  
> On Wed, Mar 30, 2011 at 3:39 PM, Pascal Deveze  wrote:
> Maybe this could solve your problem: Just add \n in the string you want to 
> display:
> printf("Please give N= \n");
> 
> Of course, this will return, but the string is displayed. This run by me 
> without the fflush().
> 
> On the other hand, do you really observe that the time of the scanf () and 
> the time to enter "N" be insignificant ?
> 
> Pascal
> 
> Meilin Bai a écrit :
>> So it means that MPI doesn't suit to interactive programming? Though we can 
>> really use fflush(stdout) to get the right order, it takes too more time, 
>> and it's said that using fflush() is not a good progrmming style in C.
>>  
>> On the other hand, in Fortran language, this situation won't exist. Maybe it 
>> is because I/O implement is a built-in part of Fortran, while in C/C++ it is 
>> realized only through function like scanf, printf, et al?
>> 
>> 
>>  
>> On Wed, Mar 30, 2011 at 2:38 AM, Prentice Bisbal  wrote:
>> On 03/29/2011 01:29 PM, Meilin Bai wrote:
>> > Dear open-mpi users:
>> >
>> > I come across a little problem when running a MPI C program compiled
>> > with Open MPI 1.4.3. A part of codes as follows:
>> >
>> > MPI_Init(&argc, &argv);
>> > MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>> > MPI_Comm_rank(MPI_COMM_WORLD, &myid);
>> > MPI_Get_processor_name(processor_name, &namelen);
>> > if (myid == 0) {
>> >  printf("Please give N= ");
>> >  //fflush(stdout);
>> >  scanf("%d", &n);
>> >  startwtime = MPI_Wtime();
>> >  }
>> >
>> > If comment out the sentence of "fflush(stdout);", it doesn't print out
>> > the message till I input an integer n. And if I add the fflush function
>> > between them, it works as expected, though comsumming time obviously.
>> >
>> > However, when I compiled it with Mpich2-1.3.2p1, without fflush function
>> > in the code, it works correctly.
>> >
>> > Can anyone know what the matter is.
>> >
>> 
>> The Open MPI Developers (Jeff, Ralph, etc) can confirm this:
>> 
>> The MPI standard doesn't have a lot of strict requirements for I/O
>> behavior like this, so implementations are allowed to buffer I/O if they
>> want. There is nothing wrong with requiring fflush(stdout) in order to
>> get the behavior you want. In fact, if you check some text books on MPI
>> programming, I'm pretty sure they recommend using fflush to minimize
>> this problem.
>> 
>> MPICH behaves differently because its developers made different design
>> choices.
>> 
>> Neither behavior is "wrong".
>> 
>> --
>> Prentice
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>> 
>> 
>> 
>> -- 
>> Meilin Bai
>> 
>> 
>> ___
>> users mailing list
>> 
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> 
> -- 
> Meilin Bai
> School of Electronic Engineering and Computer Science, Peking University
> Beijing 100871, China
> E-Mail: meilin@gmail.com
>mei...@pku.edu.cn
> MSN: meilin@hotmail.com
> Cellular: +86-1342-619-8430
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy.

Re: [OMPI users] help with hybrid openmp and mpi

[Jeff Squyres]

If you have OpenMP questions, you might want to direct those to a different 
list and/or check the relevant compiler documentation; this list is for Open 
MPI support.

Good luck.


On Mar 29, 2011, at 5:26 PM, job hunter wrote:

> hi all,
> mpiCC -openmp test.c -o test . I fixed the error I had before. I bump 
> into new question. 
> I attached my question at http://pastebin.com/63Gy854H
> thanks
> 
> 
> 
> 
> On Mon, Mar 28, 2011 at 6:46 AM, Jeff Squyres (jsquyres)  
> wrote:
> Your program is invalid for several reasons - try correcting all the compiler 
> warnings first. 
> 
> Sent from my phone. No type good. 
> 
> On Mar 27, 2011, at 10:08 PM, "job hunter"  wrote:
> 
>> Hi all,
>>I explained the problem I'm facing @  http://www.ideone.com/EGMMn
>> please help
>> thanks
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] Shared Memory Performance Problem.

[Jeff Squyres]

How many messages are you sending, and how large are they?  I.e., if your 
message passing is tiny, then the network transport may not be the bottleneck 
here.


On Mar 28, 2011, at 9:41 AM, Michele Marena wrote:

> I run ompi_info --param btl sm and this is the output
> 
>  MCA btl: parameter "btl_base_debug" (current value: "0")
>   If btl_base_debug is 1 standard debug is output, if 
> > 1 verbose debug is output
>  MCA btl: parameter "btl" (current value: )
>   Default selection set of components for the btl 
> framework ( means "use all components that can be found")
>  MCA btl: parameter "btl_base_verbose" (current value: "0")
>   Verbosity level for the btl framework (0 = no 
> verbosity)
>  MCA btl: parameter "btl_sm_free_list_num" (current value: 
> "8")
>  MCA btl: parameter "btl_sm_free_list_max" (current value: 
> "-1")
>  MCA btl: parameter "btl_sm_free_list_inc" (current value: 
> "64")
>  MCA btl: parameter "btl_sm_exclusivity" (current value: 
> "65535")
>  MCA btl: parameter "btl_sm_latency" (current value: "100")
>  MCA btl: parameter "btl_sm_max_procs" (current value: "-1")
>  MCA btl: parameter "btl_sm_sm_extra_procs" (current value: 
> "2")
>  MCA btl: parameter "btl_sm_mpool" (current value: "sm")
>  MCA btl: parameter "btl_sm_eager_limit" (current value: 
> "4096")
>  MCA btl: parameter "btl_sm_max_frag_size" (current value: 
> "32768")
>  MCA btl: parameter "btl_sm_size_of_cb_queue" (current value: 
> "128")
>  MCA btl: parameter "btl_sm_cb_lazy_free_freq" (current 
> value: "120")
>  MCA btl: parameter "btl_sm_priority" (current value: "0")
>  MCA btl: parameter "btl_base_warn_component_unused" (current 
> value: "1")
>   This parameter is used to turn on warning messages 
> when certain NICs are not used
> 
> 
> 2011/3/28 Ralph Castain 
> The fact that this exactly matches the time you measured with shared memory 
> is suspicious. My guess is that you aren't actually using shared memory at 
> all.
> 
> Does your "ompi_info" output show shared memory as being available? Jeff or 
> others may be able to give you some params that would let you check to see if 
> sm is actually being used between those procs.
> 
> 
> 
> On Mar 28, 2011, at 7:51 AM, Michele Marena wrote:
> 
>> What happens with 2 processes on the same node with tcp?
>> With --mca btl self,tcp my app runs in 23s.
>> 
>> 2011/3/28 Jeff Squyres (jsquyres) 
>> Ah, I didn't catch before that there were more variables than just tcp vs. 
>> shmem. 
>> 
>> What happens with 2 processes on the same node with tcp?
>> 
>> Eg, when both procs are on the same node, are you thrashing caches or memory?
>> 
>> Sent from my phone. No type good. 
>> 
>> On Mar 28, 2011, at 6:27 AM, "Michele Marena"  
>> wrote:
>> 
>>> However, I thank you Tim, Ralh and Jeff.
>>> My sequential application runs in 24s (wall clock time).
>>> My parallel application runs in 13s with two processes on different nodes.
>>> With shared memory, when two processes are on the same node, my app runs in 
>>> 23s.
>>> I'm not understand why.
>>> 
>>> 2011/3/28 Jeff Squyres 
>>> If your program runs faster across 3 processes, 2 of which are local to 
>>> each other, with --mca btl tcp,self compared to --mca btl tcp,sm,self, then 
>>> something is very, very strange.
>>> 
>>> Tim cites all kinds of things that can cause slowdowns, but it's still 
>>> very, very odd that simply enabling using the shared memory communications 
>>> channel in Open MPI *slows your overall application down*.
>>> 
>>> How much does your application slow down in wall clock time?  Seconds?  
>>> Minutes?  Hours?  (anything less than 1 second is in the noise)
>>> 
>>> 
>>> 
>>> On Mar 27, 2011, at 10:33 AM, Ralph Castain wrote:
>>> 
>>> >
>>> > On Mar 27, 2011, at 7:37 AM, Tim Prince wrote:
>>> >
>>> >> On 3/27/2011 2:26 AM, Michele Marena wrote:
>>> >>> Hi,
>>> >>> My application performs good without shared memory utilization, but with
>>> >>> shared memory I get performance worst than without of it.
>>> >>> Do I make a mistake? Don't I pay attention to something?
>>> >>> I know OpenMPI uses /tmp directory to allocate shared memory and it is
>>> >>> in the local filesystem.
>>> >>>
>>> >>
>>> >> I guess you mean shared memory message passing.   Among relevant 
>>> >> parameters may be the message size where your implementation switches 
>>> >> from cached copy to non-temporal (if you are on a platform where that 
>>> >> terminology is used).  If built with Intel compilers, for example, the 
>>> >> copy may be performed by intel_fast_memcpy, with a default setting which 
>>> >> uses non-temporal when the message exceeds about some p

Re: [OMPI users] OpenMPI-PGI: /usr/bin/ld: Warning: size of symbol `#' changed from # in #.o to # in #.so

[Jeff Squyres]

When I have seen this problem before, it *usually* indicated a mismatch of Open 
MPI versions that were not ABI compatible.  I.e., the application was compiled 
and linked against Open MPI version X, but then at run time, it found the 
shared libraries for Open MPI version Y -- and X is not ABI compatible with Y.

That being said:

>> Warning: size of symbol `mpi_fortran_argv_null_' changed
>> from 1 in foo.o to 8 in lib/libfoolib2.so

Doesn't sound like a version mismatch problem.  A quick check through the 
history of OMPI's source code reveals that mpi_fortran_argv_null has been a 
(char*) since way back in v1.2.x days.  So I'm curious as to how it could have 
a size of 1.

How did you compile foo.o?



On Mar 27, 2011, at 5:41 PM, Dmitry N. Mikushin wrote:

> I checked that this issue is not caused by using different compile
> options for different libraries. There is a set of libraries and
> executable compiled with mpif90, and this warning comes for
> executable's object and one of libraries...
> 
> 2011/3/25 Dmitry N. Mikushin :
>> Hi,
>> 
>> I'm wondering if anybody have seen something similar, and have you
>> succeeded to run your application compiled by openmpi-pgi-1.4.2 with
>> the following warnings:
>> 
>> /usr/bin/ld: Warning: size of symbol `mpi_fortran_errcodes_ignore_'
>> changed from 4 in foo.o to 8 in lib/libfoolib2.so
>> /usr/bin/ld: Warning: size of symbol `mpi_fortran_argv_null_' changed
>> from 1 in foo.o to 8 in lib/libfoolib2.so
>> /usr/bin/ld: Warning: alignment 16 of symbol `_mpl_message_mod_0_' in
>> lib/libfoolib1.so is smaller than 32 in foo.o
>> /usr/bin/ld: Warning: alignment 16 of symbol `_mpl_abort_mod_0_' in
>> lib/libfoolib1.so is smaller than 32 in foo.o
>> /usr/bin/ld: Warning: alignment 16 of symbol `_mpl_ioinit_mod_0_' in
>> lib/libfoolib1.so is smaller than 32 in foo.o
>> /usr/bin/ld: Warning: alignment 16 of symbol `_mpl_gatherv_mod_6_' in
>> lib/libfoolib1.so is smaller than 32 in foo.o
>> 
>> Symbols names look like being internal to OpenMPI, there was one
>> similar issue in archive back in 2006:
>> https://svn.open-mpi.org/trac/ompi/changeset/11057 could it be hit
>> again?
>> 
>> Thanks,
>> - D.
>> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI users] Fault tolerant ompi - Error: Unable to find a list of active MPIRUN processes on this machine.

[Hellmüller Roman]

Hi

I'm trying to get fault tolerant ompi running on our cluster for my 
semesterthesis.

On the login node i was successful, checkpointing works.
Since the compute nodes have different kernels, i had to compile blcr on the 
compute nodes again.  blcr on the compute nodes works. after that i installed 
openmpi (1.5.3) on the compute nodes. Letting a normal mpi program run works. 
also letting it run with -am ft-enable-cr  works, but as soon as i would like 
to take a checkpoint it crashes:

hroman@node15 ~/semesterthesis/code/code1_heat1d $ mpirun -np 4 -am 
ft-enable-cr ./heatft_mpi

hroman@node15 ~ $ ps -a
  PID TTY  TIME CMD
22488 pts/000:00:00 pbs_mom
22536 pts/000:00:00 bash
22631 pts/000:00:00 mpirun
22633 pts/000:00:03 heatft_mpi
22634 pts/000:00:03 heatft_mpi
22635 pts/000:00:03 heatft_mpi
22636 pts/000:00:03 heatft_mpi
22743 pts/100:00:00 ps

hroman@node15 ~ $ ompi-checkpoint 22631
--
Error: Unable to find a list of active MPIRUN processes on this machine.
   This could be due to one of the following:
- The PID specified (22631) is not that of an active MPIRUN.
- The session directory location could not be found/parsed.

   ompi-checkpoint attempted to find the session directory:
 /tmp//openmpi-sessions-hroman@node15_0
   Check to make sure that this directory exists while the MPIRUN
   process is running.

   Return Code: -13 (Not found)

--

I've tried it with an other application, that doesn't change anything. I also 
tried to set the checkpoint directorys in $prefix/ect/openmpi-mca-params.conf 
but that didn't seem to have any effect. however if i write errors in this file 
(smth that is no parameter eg. "hello world") it will complain, so it seems to 
read the file.
I also checked the environement variables but they seem to be ok, as far as i 
can tell.

do you have an idea where the error could be?

here 
http://n.ethz.ch/~hroman/downloads/ompi_mailinglist.tar.gz
 (40MB) you'll find the library and the build of openmpi & blcr as well as the 
env variables and the output of ompi_info. please let me know if more outputs 
are needed.

cheers
roman

Re: [OMPI users] Shared Memory Performance Problem.

[Michele Marena]

Hi Jeff,
I thank you for your help,
I've launched my app with mpiP both when two processes are on different node
and when two processes are on the same node.

The process 0 is the manager (gathers the results only), processes 1 and 2
are  workers (compute).

This is the case processes 1 and 2 are on different nodes (runs in 162s).

---
@--- MPI Time (seconds) ---
---
TaskAppTimeMPITime MPI%
   016216299.99
   1162   30.218.66
   2162   14.7 9.04
   *48620742.56
---
@--- Aggregate Time (top twenty, descending, milliseconds) 
---
Call Site   TimeApp%MPI% COV
Barrier 5   1.28e+05   26.24   61.640.00
Barrier142.3e+044.74   11.130.00
Barrier 6   2.29e+044.72   11.080.00
Barrier17   1.77e+043.658.581.41
Recv3   1.15e+042.375.580.00
Recv   30   2.26e+030.471.090.00
Recv   123080.060.150.00
Recv   262860.060.140.00
Recv   282520.050.120.00
Recv   312460.050.120.00
Isend  351110.020.050.00
Isend  341080.020.050.00
Isend  181070.020.050.00
Isend  191050.020.050.00
Isend   9   57.70.010.030.05
Isend  32   39.70.010.020.00
Barrier25   38.70.010.021.39
Isend  11   38.60.010.020.00
Recv   16   34.10.010.020.00
Recv   27   26.50.010.010.00
---
@--- Aggregate Sent Message Size (top twenty, descending, bytes) --
---
Call Site  Count  Total   Avrg  Sent%
Isend   9   4096   1.34e+08   3.28e+04  58.73
Isend  34   1200   1.85e+07   1.54e+04   8.07
Isend  35   1200   1.85e+07   1.54e+04   8.07
Isend  18   1200   1.85e+07   1.54e+04   8.07
Isend  19   1200   1.85e+07   1.54e+04   8.07
Isend  32240   3.69e+06   1.54e+04   1.61
Isend  11240   3.69e+06   1.54e+04   1.61
Isend  15180   3.44e+06   1.91e+04   1.51
Isend  33 61  2e+06   3.28e+04   0.87
Isend  10 61  2e+06   3.28e+04   0.87
Isend  29 61  2e+06   3.28e+04   0.87
Isend  22 61  2e+06   3.28e+04   0.87
Isend  37180   1.72e+06   9.57e+03   0.75
Isend  24  2 16  8   0.00
Isend  20  2 16  8   0.00
Send8  1  4  4   0.00
Send1  1  4  4   0.00

The case when processes 1 and 2 are on the same node (runs in 260s).
---
@--- MPI Time (seconds) ---
---
TaskAppTimeMPITime MPI%
   026026099.99
   1260   39.715.29
   2260   26.410.17
   *77932641.82

---
@--- Aggregate Time (top twenty, descending, milliseconds) 
---
Call Site   TimeApp%MPI% COV
Barrier 5   2.23e+05   28.64   68.500.00
Barrier 6   2.49e+043.207.660.00
Barrier14   2.31e+042.967.090.00
Recv   28   1.35e+041.734.140.00
Recv   16   1.32e+041.704.060.00
Barrier17   1.22e+041.563.741.41
Recv3   1.16e+041.483.550.00
Recv   26   1.67e+030.210.510.00
Recv   309400.120.290.00
Recv   

Re: [OMPI users] Shared Memory Performance Problem.

[Eugene Loh]

Michele Marena wrote:
I've launched my app with mpiP both when two processes are
on different node and when two processes are on the same node.
  
  
  The process 0 is the manager (gathers the results only),
processes 1 and 2 are  workers (compute).
  
  
  This is the case processes 1 and 2 are on different nodes (runs
in 162s).
  
  @--- MPI Time (seconds)
---
  Task    AppTime    MPITime     MPI%
     0        162        162    99.99
     1        162       30.2    18.66
     2        162       14.7     9.04
     *        486        207    42.56
  
  
  The case when processes 1 and 2 are on the same node (runs in
260s).
  
  @--- MPI Time (seconds)
---
  Task    AppTime    MPITime     MPI%
     0        260        260    99.99
     1        260       39.7    15.29
     2        260       26.4    10.17
     *        779        326    41.82
  
  
  
  I think there's a contention problem on the memory bus.
  

Right.  Process 0 spends all its time in MPI, presumably waiting on
workers.  The workers spend about the same amount of time on MPI
regardless of whether they're placed together or not.  The big
difference is that the workers are much slower in non-MPI tasks when
they're located on the same node.  The issue has little to do with
MPI.  The workers are hogging local resources and work faster when
placed on different nodes.

  
  However, the message size is 4096 * sizeof(double). Maybe I are
wrong in this point. Is the message size too huge for shared memory?
  

No.  That's not very large at all.

  
  
  

>>> On Mar 27, 2011, at 10:33 AM, Ralph
Castain wrote:
>>>
>>> >http://www.open-mpi.org/faq/?category=perftools

  
  
  

Re: [OMPI users] Shared Memory Performance Problem.

[Tim Prince]

On 3/30/2011 10:08 AM, Eugene Loh wrote:

Michele Marena wrote:

I've launched my app with mpiP both when two processes are on
different node and when two processes are on the same node.

The process 0 is the manager (gathers the results only), processes 1
and 2 are workers (compute).

This is the case processes 1 and 2 are on different nodes (runs in 162s).
@--- MPI Time (seconds)
---
Task AppTime MPITime MPI%
0 162 162 99.99
1 162 30.2 18.66
2 162 14.7 9.04
* 486 207 42.56

The case when processes 1 and 2 are on the same node (runs in 260s).
@--- MPI Time (seconds)
---
Task AppTime MPITime MPI%
0 260 260 99.99
1 260 39.7 15.29
2 260 26.4 10.17
* 779 326 41.82

I think there's a contention problem on the memory bus.

Right. Process 0 spends all its time in MPI, presumably waiting on
workers. The workers spend about the same amount of time on MPI
regardless of whether they're placed together or not. The big difference
is that the workers are much slower in non-MPI tasks when they're
located on the same node. The issue has little to do with MPI. The
workers are hogging local resources and work faster when placed on
different nodes.

However, the message size is 4096 * sizeof(double). Maybe I are wrong
in this point. Is the message size too huge for shared memory?

No. That's not very large at all.


Not even large enough to expect the non-temporal storage issue about 
cache eviction to arise.



--
Tim Prince

[OMPI users] mpi problems,

[Nehemiah Dacres]

I am trying to figure out why my jobs aren't getting distributed and need
some help. I have an install of sun cluster tools on Rockscluster 5.2
(essentially centos4u2). this user's account has its home dir shared via
nfs. I am getting some strange errors. here's an example run


[jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/mpirun -np 3 -hostfile list
./job2.sh
bash: /opt/SUNWhpc/HPC8.2.1c/sun/bin/orted: No such file or directory
--
A daemon (pid 20362) died unexpectedly with status 127 while attempting
to launch so we are aborting.

There may be more information reported by the environment (see above).

This may be because the daemon was unable to find all the needed shared
libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
location of the shared libraries on the remote nodes and this will
automatically be forwarded to the remote nodes.
--
--
mpirun noticed that the job aborted, but has no info as to the process
that caused that situation.
--
mpirun: clean termination accomplished

[jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/
bin/examples/   instrument/ man/
etc/include/lib/share/
[jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/orte
orte-clean  orted   orte-ioforte-ps orterun
[jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/orted
[therock.slu.loc:20365] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in
file runtime/orte_init.c at line 125
--
It looks like orte_init failed for some reason; your parallel process is
likely to abort.  There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems.  This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):

  orte_ess_base_select failed
  --> Returned value Not found (-13) instead of ORTE_SUCCESS
--
[therock.slu.loc:20365] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in
file orted/orted_main.c at line 325
[jian@therock ~]$


-- 
Nehemiah I. Dacres
System Administrator
Advanced Technology Group Saint Louis University

Re: [OMPI users] mpi problems,

[David Zhang]

As one of the error message suggests, you need to add the openmpi library to
your LD_LIBRARY_PATH to all your nodes.

On Wed, Mar 30, 2011 at 1:24 PM, Nehemiah Dacres  wrote:

> I am trying to figure out why my jobs aren't getting distributed and need
> some help. I have an install of sun cluster tools on Rockscluster 5.2
> (essentially centos4u2). this user's account has its home dir shared via
> nfs. I am getting some strange errors. here's an example run
>
>
> [jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/mpirun -np 3 -hostfile
> list ./job2.sh
> bash: /opt/SUNWhpc/HPC8.2.1c/sun/bin/orted: No such file or directory
> --
> A daemon (pid 20362) died unexpectedly with status 127 while attempting
> to launch so we are aborting.
>
> There may be more information reported by the environment (see above).
>
> This may be because the daemon was unable to find all the needed shared
> libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
> location of the shared libraries on the remote nodes and this will
> automatically be forwarded to the remote nodes.
> --
> --
> mpirun noticed that the job aborted, but has no info as to the process
> that caused that situation.
> --
> mpirun: clean termination accomplished
>
> [jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/
> bin/examples/   instrument/ man/
> etc/include/lib/share/
> [jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/orte
> orte-clean  orted   orte-ioforte-ps orterun
> [jian@therock ~]$ /opt/SUNWhpc/HPC8.2.1c/sun/bin/orted
> [therock.slu.loc:20365] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in
> file runtime/orte_init.c at line 125
> --
> It looks like orte_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during orte_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>   orte_ess_base_select failed
>   --> Returned value Not found (-13) instead of ORTE_SUCCESS
> --
> [therock.slu.loc:20365] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in
> file orted/orted_main.c at line 325
> [jian@therock ~]$
>
>
> --
> Nehemiah I. Dacres
> System Administrator
> Advanced Technology Group Saint Louis University
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
David Zhang
University of California, San Diego