Re: [OMPI users] OpenMPI providing rank?

[Nysal Jan]

OMPI_COMM_WORLD_RANK can be used to get the MPI rank. For other environment
variables -
http://www.open-mpi.org/faq/?category=running#mpi-environmental-variables
For processor affinity see this FAQ entry -
http://www.open-mpi.org/faq/?category=all#using-paffinity

--Nysal

On Wed, Jul 28, 2010 at 9:04 AM, Yves Caniou wrote:

> Hi,
>
> I have some performance issue on a parallel machine composed of nodes of 16
> procs each. The application is launched on multiple of 16 procs for given
> numbers of nodes.
> I was told by people using MX MPI with this machine to attach a script to
> mpiexec, which 'numactl' things, in order to make the execution performance
> stable.
>
> Looking on the faq (the oldest one is for OpenMPI v1.3?), I saw that maybe
> the
> solution would be for me to use the --mca mpi_paffinity_alone 1
> Is that correct? -- BTW, I have both memory and processor affinity:
> >ompi_info | grep affinity
>   MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.2)
>   MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.2)
>   MCA maffinity: libnuma (MCA v2.0, API v2.0, Component v1.4.2)
> Does it handle memory too, or do I have to use another option like
> --mca mpi_maffinity 1?
>
> Still, I would like to test the numactl solution. Does OpenMPI provide an
> equivalent to $MXMPI_ID which gives at least gives the NODE on which a
> process is launched by OpenMPI, so that I can adapt the script that was
> given
> to me?
>
> Tkx.
>
> .Yves.
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] OpenMPI providing rank?

[Yves Caniou]

Le Wednesday 28 July 2010 06:03:21 Nysal Jan, vous avez écrit :
> OMPI_COMM_WORLD_RANK can be used to get the MPI rank. For other environment
> variables -
> http://www.open-mpi.org/faq/?category=running#mpi-environmental-variables

Are processes affected to nodes sequentially, so that I can get the NODE 
number from $OMPI_COMM_WORLD_RANK modulo the number of proc per node?

> For processor affinity see this FAQ entry -
> http://www.open-mpi.org/faq/?category=all#using-paffinity

Thank you, but that's where I had the information that I put in my previous 
mail, so it doesn't answer to my question.

.Yves.

> --Nysal
>
> On Wed, Jul 28, 2010 at 9:04 AM, Yves Caniou wrote:
> > Hi,
> >
> > I have some performance issue on a parallel machine composed of nodes of
> > 16 procs each. The application is launched on multiple of 16 procs for
> > given numbers of nodes.
> > I was told by people using MX MPI with this machine to attach a script to
> > mpiexec, which 'numactl' things, in order to make the execution
> > performance stable.
> >
> > Looking on the faq (the oldest one is for OpenMPI v1.3?), I saw that
> > maybe the
> > solution would be for me to use the --mca mpi_paffinity_alone 1
> >
> > Is that correct? -- BTW, I have both memory and processor affinity:
> > >ompi_info | grep affinity
> >
> >   MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.2)
> >   MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.2)
> >   MCA maffinity: libnuma (MCA v2.0, API v2.0, Component v1.4.2)
> > Does it handle memory too, or do I have to use another option like
> > --mca mpi_maffinity 1?
> >
> > Still, I would like to test the numactl solution. Does OpenMPI provide an
> > equivalent to $MXMPI_ID which gives at least gives the NODE on which a
> > process is launched by OpenMPI, so that I can adapt the script that was
> > given
> > to me?
> >
> > Tkx.
> >
> > .Yves.
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users



-- 
Yves Caniou
Associate Professor at Université Lyon 1,
Member of the team project INRIA GRAAL in the LIP ENS-Lyon,
Délégation CNRS in Japan French Laboratory of Informatics (JFLI),
  * in Information Technology Center, The University of Tokyo,
2-11-16 Yayoi, Bunkyo-ku, Tokyo 113-8658, Japan
tel: +81-3-5841-0540
  * in National Institute of Informatics
2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan
tel: +81-3-4212-2412 
http://graal.ens-lyon.fr/~ycaniou/

Re: [OMPI users] Dynamic processes connection and segfault on MPI_Comm_accept

[Grzegorz Maj]

I've attached gdb to the client which has just connected to the grid.
Its bt is almost exactly the same as the server's one:
#0  0x428066d7 in sched_yield () from /lib/libc.so.6
#1  0x00933cbf in opal_progress () at ../../opal/runtime/opal_progress.c:220
#2  0x00d460b8 in opal_condition_wait (c=0xdc3160, m=0xdc31a0) at
../../opal/threads/condition.h:99
#3  0x00d463cc in ompi_request_default_wait_all (count=2,
requests=0xff8a36d0, statuses=0x0) at
../../ompi/request/req_wait.c:262
#4  0x00a1431f in mca_coll_inter_allgatherv_inter (sbuf=0xff8a3794,
scount=1, sdtype=0x8049400, rbuf=0xff8a3750, rcounts=0x80948e0,
disps=0x8093938, rdtype=0x8049400, comm=0x8094fb8, module=0x80954a0)
at ../../../../../ompi/mca/coll/inter/coll_inter_allgatherv.c:127
#5  0x00d3198f in ompi_comm_determine_first (intercomm=0x8094fb8,
high=1) at ../../ompi/communicator/comm.c:1199
#6  0x00d75833 in PMPI_Intercomm_merge (intercomm=0x8094fb8, high=1,
newcomm=0xff8a4c00) at pintercomm_merge.c:84
#7  0x08048a16 in main (argc=892352312, argv=0x32323038) at client.c:28

I've tried both scenarios described: when hangs a client connecting
from machines B and C. In both cases bt looks the same.
How does it look like?
Shall I repost that using a different subject as Ralph suggested?

Regards,
Grzegorz



2010/7/27 Edgar Gabriel :
> based on your output shown here, there is absolutely nothing wrong
> (yet). Both processes are in the same function and do what they are
> supposed to do.
>
> However, I am fairly sure that the client process bt that you show is
> already part of current_intracomm. Could you try to create a bt of the
> process that is not yet part of current_intracomm (If I understand your
> code correctly, the intercommunicator is n-1 configuration, with each
> client process being part of n after the intercomm_merge). It would be
> interesting to see where that process is...
>
> Thanks
> Edgar
>
> On 7/27/2010 1:42 PM, Ralph Castain wrote:
>> This slides outside of my purview - I would suggest you post this question 
>> with a different subject line specifically mentioning failure of 
>> intercomm_merge to work so it attracts the attention of those with knowledge 
>> of that area.
>>
>>
>> On Jul 27, 2010, at 9:30 AM, Grzegorz Maj wrote:
>>
>>> So now I have a new question.
>>> When I run my server and a lot of clients on the same machine,
>>> everything looks fine.
>>>
>>> But when I try to run the clients on several machines the most
>>> frequent scenario is:
>>> * server is stared on machine A
>>> * X (= 1, 4, 10, ..) clients are started on machine B and they connect
>>> successfully
>>> * the first client starting on machine C connects successfully to the
>>> server, but the whole grid hangs on MPI_Comm_merge (all the processes
>>> from intercommunicator get there).
>>>
>>> As I said it's the most frequent scenario. Sometimes I can connect the
>>> clients from several machines. Sometimes it hangs (always on
>>> MPI_Comm_merge) when connecting the clients from machine B.
>>> The interesting thing is, that if before MPI_Comm_merge I send a dummy
>>> message on the intercommunicator from process rank 0 in one group to
>>> process rank 0 in the other one, it will not hang on MPI_Comm_merge.
>>>
>>> I've tried both versions with and without the first patch (ompi-server
>>> as orted) but it doesn't change the behavior.
>>>
>>> I've attached gdb to my server, this is bt:
>>> #0  0xe410 in __kernel_vsyscall ()
>>> #1  0x00637afc in sched_yield () from /lib/libc.so.6
>>> #2  0xf7e8ce31 in opal_progress () at ../../opal/runtime/opal_progress.c:220
>>> #3  0xf7f60ad4 in opal_condition_wait (c=0xf7fd7dc0, m=0xf7fd7e00) at
>>> ../../opal/threads/condition.h:99
>>> #4  0xf7f60dee in ompi_request_default_wait_all (count=2,
>>> requests=0xff8d7754, statuses=0x0) at
>>> ../../ompi/request/req_wait.c:262
>>> #5  0xf7d3e221 in mca_coll_inter_allgatherv_inter (sbuf=0xff8d7824,
>>> scount=1, sdtype=0x8049200, rbuf=0xff8d77e0, rcounts=0x9783df8,
>>> disps=0x9755520, rdtype=0x8049200, comm=0x978c2a8, module=0x9794b08)
>>>    at ../../../../../ompi/mca/coll/inter/coll_inter_allgatherv.c:127
>>> #6  0xf7f4c615 in ompi_comm_determine_first (intercomm=0x978c2a8,
>>> high=0) at ../../ompi/communicator/comm.c:1199
>>> #7  0xf7f8d1d9 in PMPI_Intercomm_merge (intercomm=0x978c2a8, high=0,
>>> newcomm=0xff8d78c0) at pintercomm_merge.c:84
>>> #8  0x0804893c in main (argc=Cannot access memory at address 0xf
>>> ) at server.c:50
>>>
>>> And this is bt from one of the clients:
>>> #0  0xe410 in __kernel_vsyscall ()
>>> #1  0x0064993b in poll () from /lib/libc.so.6
>>> #2  0xf7de027f in poll_dispatch (base=0x8643fb8, arg=0x86442d8,
>>> tv=0xff82299c) at ../../../opal/event/poll.c:168
>>> #3  0xf7dde4b2 in opal_event_base_loop (base=0x8643fb8, flags=2) at
>>> ../../../opal/event/event.c:807
>>> #4  0xf7dde34f in opal_event_loop (flags=2) at 
>>> ../../../opal/event/event.c:730
>>> #5  0xf7dcfc77 in opal_progress () at ../../opal/runtime/opal_progress.c:1

Re: [OMPI users] OpenMPI providing rank?

[Ralph Castain]

On Jul 27, 2010, at 11:18 PM, Yves Caniou wrote:

> Le Wednesday 28 July 2010 06:03:21 Nysal Jan, vous avez écrit :
>> OMPI_COMM_WORLD_RANK can be used to get the MPI rank. For other environment
>> variables -
>> http://www.open-mpi.org/faq/?category=running#mpi-environmental-variables
> 
> Are processes affected to nodes sequentially, so that I can get the NODE 
> number from $OMPI_COMM_WORLD_RANK modulo the number of proc per node?

By default, yes. However, you can select alternative mapping methods.

Or...you could just use the mpirun cmd line option to report the binding of 
each process as it is started :-)

Do "mpirun -h" to see all the options. The one you want is --report-bindings

> 
>> For processor affinity see this FAQ entry -
>> http://www.open-mpi.org/faq/?category=all#using-paffinity
> 
> Thank you, but that's where I had the information that I put in my previous 
> mail, so it doesn't answer to my question.

Memory affinity is taken care of under-the-covers when paffinity is active. No 
other options are required.


> 
> .Yves.
> 
>> --Nysal
>> 
>> On Wed, Jul 28, 2010 at 9:04 AM, Yves Caniou wrote:
>>> Hi,
>>> 
>>> I have some performance issue on a parallel machine composed of nodes of
>>> 16 procs each. The application is launched on multiple of 16 procs for
>>> given numbers of nodes.
>>> I was told by people using MX MPI with this machine to attach a script to
>>> mpiexec, which 'numactl' things, in order to make the execution
>>> performance stable.
>>> 
>>> Looking on the faq (the oldest one is for OpenMPI v1.3?), I saw that
>>> maybe the
>>> solution would be for me to use the --mca mpi_paffinity_alone 1
>>> 
>>> Is that correct? -- BTW, I have both memory and processor affinity:
 ompi_info | grep affinity
>>> 
>>>  MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.2)
>>>  MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4.2)
>>>  MCA maffinity: libnuma (MCA v2.0, API v2.0, Component v1.4.2)
>>> Does it handle memory too, or do I have to use another option like
>>> --mca mpi_maffinity 1?
>>> 
>>> Still, I would like to test the numactl solution. Does OpenMPI provide an
>>> equivalent to $MXMPI_ID which gives at least gives the NODE on which a
>>> process is launched by OpenMPI, so that I can adapt the script that was
>>> given
>>> to me?
>>> 
>>> Tkx.
>>> 
>>> .Yves.
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
> 
> 
> -- 
> Yves Caniou
> Associate Professor at Université Lyon 1,
> Member of the team project INRIA GRAAL in the LIP ENS-Lyon,
> Délégation CNRS in Japan French Laboratory of Informatics (JFLI),
>  * in Information Technology Center, The University of Tokyo,
>2-11-16 Yayoi, Bunkyo-ku, Tokyo 113-8658, Japan
>tel: +81-3-5841-0540
>  * in National Institute of Informatics
>2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan
>tel: +81-3-4212-2412 
> http://graal.ens-lyon.fr/~ycaniou/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Processes stuck after MPI_Waitall() in 1.4.1

[Terry Dontje]

Here are a couple other suggestions:

1.  Have you tried your code with using the TCP btl just to make sure 
this might not be a general algorithm issue with the collective?


2.  While using the openib btl you may want to try things with rdma 
turned off by using the following parameters to mpirun:
-mca btl_openib_use_eager_rdma 0 -mca btl_openib_max_eager_rdma 0 -mca 
btl_openib_flags 1


3.  While using the openib btl you may want to try things while bumping 
up the rendezvous limit to see  if eliminating rendezvous messages helps 
things (others on the list is there an easier way to do this?).  Set the 
following parameters raising the 8192 and 2048 values:
-mca btl_openib_receive_queues "P,8192" -mca btl_openib_max_send_size 
8192 -mca btl_openib_eager_limit 8192 -mca btl_openib_rndv_eager_limit 2048


4.  You mayb also want to try and see if the basic collective algorithm 
work instead of using the tuned, which is the default I believe, by 
setting  "-mca coll_basic_priority 100".  The idea here is to determine 
if the tuned collective itself is tickling the issue.


--td

Terry Dontje wrote:
With this earlier failure do you know how many message may have been 
transferred between the two processes?  Is there a way to narrow this 
down to a small piece of code?  Do you have totalview or ddt at your 
disposal?


--td

Brian Smith wrote:

Also, the application I'm having trouble with appears to work fine with
MVAPICH2 1.4.1, if that is any help.

-Brian

On Tue, 2010-07-27 at 10:48 -0400, Terry Dontje wrote:
  

Can you try a simple point-to-point program?

--td

Brian Smith wrote: 


After running on two processors across two nodes, the problem occurs
much earlier during execution:

(gdb) bt
#0  opal_sys_timer_get_cycles ()
at ../opal/include/opal/sys/amd64/timer.h:46
#1  opal_timer_base_get_cycles ()
at ../opal/mca/timer/linux/timer_linux.h:31
#2  opal_progress () at runtime/opal_progress.c:181
#3  0x2b4bc3c00215 in opal_condition_wait (count=2,
requests=0x7fff33372480, statuses=0x7fff33372450)
at ../opal/threads/condition.h:99
#4  ompi_request_default_wait_all (count=2, requests=0x7fff33372480,
statuses=0x7fff33372450) at request/req_wait.c:262
#5  0x2b4bc3c915b7 in ompi_coll_tuned_sendrecv_actual
(sendbuf=0x2aaad11dfaf0, scount=117692, 
sdatatype=0x2b4bc3fa9ea0, dest=1, stag=-13, recvbuf=out>, rcount=117692, 
rdatatype=0x2b4bc3fa9ea0, source=1, rtag=-13, comm=0x12cd98c0,

status=0x0) at coll_tuned_util.c:55
#6  0x2b4bc3c982db in ompi_coll_tuned_sendrecv (sbuf=0x2aaad10f9d10,
scount=117692, sdtype=0x2b4bc3fa9ea0, 
rbuf=0x2aaae104d010, rcount=117692, rdtype=0x2b4bc3fa9ea0,

comm=0x12cd98c0, module=0x12cda340)
at coll_tuned_util.h:60
#7  ompi_coll_tuned_alltoall_intra_two_procs (sbuf=0x2aaad10f9d10,
scount=117692, sdtype=0x2b4bc3fa9ea0, 
rbuf=0x2aaae104d010, rcount=117692, rdtype=0x2b4bc3fa9ea0,

comm=0x12cd98c0, module=0x12cda340)
at coll_tuned_alltoall.c:432
#8  0x2b4bc3c1b71f in PMPI_Alltoall (sendbuf=0x2aaad10f9d10,
sendcount=117692, sendtype=0x2b4bc3fa9ea0, 
recvbuf=0x2aaae104d010, recvcount=117692, recvtype=0x2b4bc3fa9ea0,

comm=0x12cd98c0) at palltoall.c:84
#9  0x2b4bc399cc86 in mpi_alltoall_f (sendbuf=0x2aaad10f9d10 "Z\n
\271\356\023\254\271?", sendcount=0x7fff33372688, 
sendtype=, recvbuf=0x2aaae104d010 "",
recvcount=0x7fff3337268c, recvtype=0xb67490, 
comm=0x12d9d20, ierr=0x7fff33372690) at palltoall_f.c:76

#10 0x004613b8 in m_alltoall_z_ ()
#11 0x004ec55f in redis_pw_ ()
#12 0x005643d0 in choleski_mp_orthch_ ()
#13 0x0043fbba in MAIN__ ()
#14 0x0042f15c in main ()

On Tue, 2010-07-27 at 06:14 -0400, Terry Dontje wrote:
  
  

A clarification from your previous email, you had your code working
with OMPI 1.4.1 but an older version of OFED?  Then you upgraded to
OFED 1.4 and things stopped working?  Sounds like your current system
is set up with OMPI 1.4.2 and OFED 1.5.  Anyways, I am a little
confused as to when things might have actually broke.

My first guess would be something may be wrong with the OFED setup.
Have checked the status of your ib devices with ibv_devinfo?  Have you
ran any of the OFED rc tests like ibv_rc_pingpong?  


If the above seems ok have you tried to run a simpler OMPI test like
connectivity?  I would see if a simple np=2 run spanning across two
nodes fails?

What OS distribution and version you are running on?

--td
Brian Smith wrote: 



In case my previous e-mail is too vague for anyone to address, here's a
backtrace from my application.  This version, compiled with Intel
11.1.064 (OpenMPI 1.4.2 w/ gcc 4.4.2), hangs during MPI_Alltoall
instead.  Running on 16 CPUs, Opteron 2427, Mellanox Technologies
MT25418 w/ OFED 1.5

strace on all ranks repeatedly shows:
poll([{fd=4, events=POLLIN}, {fd=5, events=POLLIN}, {fd=6,
events=POLLIN}, {fd=7, events=POLLIN}, {fd=10, events=POLLIN}, {fd=22,
events=POLLIN}, {fd=23, events=POLLIN}], 7, 0) = 0 (Timeout

Re: [OMPI users] openMPI shared with NFS, but says different version

[Jeff Squyres]

This issue is usually caused by installing one version of Open MPI over an 
older version:

http://www.open-mpi.org/faq/?category=building#install-overwrite


On Jul 27, 2010, at 10:35 PM, Cristobal Navarro wrote:

> 
> On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa  wrote:
> Hi Cristobal
> 
> Does it run only on the head node alone?
> (Fuego? Agua? Acatenango?)
> Try to put only the head node on the hostfile and execute with mpiexec.
> --> i will try only with the head node, and post results back 
> This may help sort out what is going on.
> Hopefully it will run on the head node.
> 
> Also, do you have Infinband connecting the nodes?
> The error messages refer to the openib btl (i.e. Infiniband),
> and complains of
> 
> no we are just using normal network 100MBit/s , since i am just testing yet.
> 
> "perhaps a missing symbol, or compiled for a different
> version of Open MPI?".
> It sounds as a mixup of versions/builds.
> 
> --> i agree, somewhere there must be the remains of the older version 
> 
> Did you configure/build OpenMPI from source, or did you install
> it with apt-get?
> It may be easier/less confusing to install from source.
> If you did, what configure options did you use?
> 
> -->i installed from source, 
> ./configure --prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid 
> --disable--static 
> 
> Also, as for the OpenMPI runtime environment,
> it is not enough to set it on
> the command line, because it will be effective only on the head node.
> You need to either add them to the PATH and LD_LIBRARY_PATH
> on your .bashrc/.cshrc files (assuming these files and your home directory 
> are *also* shared with the nodes via NFS),
> or use the --prefix option of mpiexec to point to the OpenMPI main directory.
> 
> yes, all nodes have their PATH and LD_LIBRARY_PATH set up properly inside the 
> login scripts ( .bashrc in my case  ) 
> 
> Needless to say, you need to check and ensure that the OpenMPI directory (and 
> maybe your home directory, and your work directory) is (are)
> really mounted on the nodes.
> 
> --> yes, doublechecked that they are 
> 
> I hope this helps,
> 
> --> thanks really! 
> 
> Gus Correa
> 
> Update: i just reinstalled openMPI, with the same parameters, and it seems 
> that the problem has gone, i couldnt test entirely but when i get back to lab 
> ill confirm. 
> 
> best regards! 
> Cristobal
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_Allreduce on local machine

[Jeff Squyres]

On Jul 27, 2010, at 11:21 AM, Hugo Gagnon wrote:

> I appreciate your replies but my question has to do with the function
> MPI_Allreduce of OpenMPI built on a Mac OSX 10.6 with ifort (intel
> fortran compiler).

The implication I was going for was that if you were using MPI_DOUBLE_PRECISION 
with a data buffer that wasn't actually double precision, Bad Things would 
happen inside the allreduce because OMPI would likely read/write beyond the end 
of your buffer.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_Allreduce on local machine

[Jeff Squyres]

On Jul 27, 2010, at 4:19 PM, Gus Correa wrote:

> Is there a simple way to check the number of bytes associated to each
> MPI basic type of OpenMPI on a specific machine (or machine+compiler)?
> 
> Something that would come out easily, say, from ompi_info?

Not via ompi_info, but the MPI function MPI_GET_EXTENT will tell you the 
datatype's size.

-
[4:54] svbu-mpi:~/mpi % cat extent.f90
  program main
  use mpi
  implicit none
  integer ierr, ext

  call MPI_INIT(ierr)
  call MPI_TYPE_EXTENT(MPI_DOUBLE_PRECISION, ext, ierr)
  print *, 'Type extent of DOUBLE_PREC is', ext
  call MPI_FINALIZE(ierr)

  end
[4:54] svbu-mpi:~/mpi % mpif90 extent.f90 -o extent -g
[4:54] svbu-mpi:~/mpi % ./extent
 Type extent of DOUBLE_PREC is   8
[4:54] svbu-mpi:~/mpi % 
-

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OpenMPI providing rank?

[Yves Caniou]

Le Wednesday 28 July 2010 11:34:13 Ralph Castain, vous avez écrit :
> On Jul 27, 2010, at 11:18 PM, Yves Caniou wrote:
> > Le Wednesday 28 July 2010 06:03:21 Nysal Jan, vous avez écrit :
> >> OMPI_COMM_WORLD_RANK can be used to get the MPI rank. For other
> >> environment variables -
> >> http://www.open-mpi.org/faq/?category=running#mpi-environmental-variable
> >>s
> >
> > Are processes affected to nodes sequentially, so that I can get the NODE
> > number from $OMPI_COMM_WORLD_RANK modulo the number of proc per node?
>
> By default, yes. However, you can select alternative mapping methods.
>
> Or...you could just use the mpirun cmd line option to report the binding of
> each process as it is started :-)
>
> Do "mpirun -h" to see all the options. The one you want is
> --report-bindings

It reports to stderr, so the $OMPI_COMM_WORLD_RANK modulo the number of proc 
per nodes seems more appropriate for what I need, right?

So is the following valid to put memory affinity?

script.sh:
  MYRANK=$OMPI_COMM_WORLD_RANK
  MYVAL=$(expr $MYRANK / 4)
  NODE=$(expr $MYVAL % 4)
  numactl --cpunodebind=$NODE --membind=$NODE $@

mpiexec ./script.sh -n 128 myappli myparam

> >> For processor affinity see this FAQ entry -
> >> http://www.open-mpi.org/faq/?category=all#using-paffinity
> >
> > Thank you, but that's where I had the information that I put in my
> > previous mail, so it doesn't answer to my question.
>
> Memory affinity is taken care of under-the-covers when paffinity is active.
> No other options are required.

Which is better: using this option, or the cmd line with numactl (if it 
works)? What is the difference?

Tkx.

.Yves.

> > .Yves.
> >
> >> --Nysal
> >>
> >> On Wed, Jul 28, 2010 at 9:04 AM, Yves Caniou 
wrote:
> >>> Hi,
> >>>
> >>> I have some performance issue on a parallel machine composed of nodes
> >>> of 16 procs each. The application is launched on multiple of 16 procs
> >>> for given numbers of nodes.
> >>> I was told by people using MX MPI with this machine to attach a script
> >>> to mpiexec, which 'numactl' things, in order to make the execution
> >>> performance stable.
> >>>
> >>> Looking on the faq (the oldest one is for OpenMPI v1.3?), I saw that
> >>> maybe the
> >>> solution would be for me to use the --mca mpi_paffinity_alone 1
> >>>
> >>> Is that correct? -- BTW, I have both memory and processor affinity:
>  ompi_info | grep affinity
> >>>
> >>>  MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.2)
> >>>  MCA maffinity: first_use (MCA v2.0, API v2.0, Component
> >>> v1.4.2) MCA maffinity: libnuma (MCA v2.0, API v2.0, Component v1.4.2)
> >>> Does it handle memory too, or do I have to use another option like
> >>> --mca mpi_maffinity 1?
> >>>
> >>> Still, I would like to test the numactl solution. Does OpenMPI provide
> >>> an equivalent to $MXMPI_ID which gives at least gives the NODE on which
> >>> a process is launched by OpenMPI, so that I can adapt the script that
> >>> was given
> >>> to me?
> >>>
> >>> Tkx.
> >>>
> >>> .Yves.
> >>> ___
> >>> users mailing list
> >>> us...@open-mpi.org
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Dynamic processes connection and segfault on MPI_Comm_accept

[Edgar Gabriel]

hm, this looks actually correct. The question now basically is, why the
intermediate hand-shake by the processes with rank 0 on the
inter-communicator is not finishing.
I am wandering whether this could be related to a problem reported in
another thread (Processes stuck after MPI_Waitall() in 1.4.1)?

http://www.open-mpi.org/community/lists/users/2010/07/13720.php




On 7/28/2010 4:01 AM, Grzegorz Maj wrote:
> I've attached gdb to the client which has just connected to the grid.
> Its bt is almost exactly the same as the server's one:
> #0  0x428066d7 in sched_yield () from /lib/libc.so.6
> #1  0x00933cbf in opal_progress () at ../../opal/runtime/opal_progress.c:220
> #2  0x00d460b8 in opal_condition_wait (c=0xdc3160, m=0xdc31a0) at
> ../../opal/threads/condition.h:99
> #3  0x00d463cc in ompi_request_default_wait_all (count=2,
> requests=0xff8a36d0, statuses=0x0) at
> ../../ompi/request/req_wait.c:262
> #4  0x00a1431f in mca_coll_inter_allgatherv_inter (sbuf=0xff8a3794,
> scount=1, sdtype=0x8049400, rbuf=0xff8a3750, rcounts=0x80948e0,
> disps=0x8093938, rdtype=0x8049400, comm=0x8094fb8, module=0x80954a0)
> at ../../../../../ompi/mca/coll/inter/coll_inter_allgatherv.c:127
> #5  0x00d3198f in ompi_comm_determine_first (intercomm=0x8094fb8,
> high=1) at ../../ompi/communicator/comm.c:1199
> #6  0x00d75833 in PMPI_Intercomm_merge (intercomm=0x8094fb8, high=1,
> newcomm=0xff8a4c00) at pintercomm_merge.c:84
> #7  0x08048a16 in main (argc=892352312, argv=0x32323038) at client.c:28
> 
> I've tried both scenarios described: when hangs a client connecting
> from machines B and C. In both cases bt looks the same.
> How does it look like?
> Shall I repost that using a different subject as Ralph suggested?
> 
> Regards,
> Grzegorz
> 
> 
> 
> 2010/7/27 Edgar Gabriel :
>> based on your output shown here, there is absolutely nothing wrong
>> (yet). Both processes are in the same function and do what they are
>> supposed to do.
>>
>> However, I am fairly sure that the client process bt that you show is
>> already part of current_intracomm. Could you try to create a bt of the
>> process that is not yet part of current_intracomm (If I understand your
>> code correctly, the intercommunicator is n-1 configuration, with each
>> client process being part of n after the intercomm_merge). It would be
>> interesting to see where that process is...
>>
>> Thanks
>> Edgar
>>
>> On 7/27/2010 1:42 PM, Ralph Castain wrote:
>>> This slides outside of my purview - I would suggest you post this question 
>>> with a different subject line specifically mentioning failure of 
>>> intercomm_merge to work so it attracts the attention of those with 
>>> knowledge of that area.
>>>
>>>
>>> On Jul 27, 2010, at 9:30 AM, Grzegorz Maj wrote:
>>>
 So now I have a new question.
 When I run my server and a lot of clients on the same machine,
 everything looks fine.

 But when I try to run the clients on several machines the most
 frequent scenario is:
 * server is stared on machine A
 * X (= 1, 4, 10, ..) clients are started on machine B and they connect
 successfully
 * the first client starting on machine C connects successfully to the
 server, but the whole grid hangs on MPI_Comm_merge (all the processes
 from intercommunicator get there).

 As I said it's the most frequent scenario. Sometimes I can connect the
 clients from several machines. Sometimes it hangs (always on
 MPI_Comm_merge) when connecting the clients from machine B.
 The interesting thing is, that if before MPI_Comm_merge I send a dummy
 message on the intercommunicator from process rank 0 in one group to
 process rank 0 in the other one, it will not hang on MPI_Comm_merge.

 I've tried both versions with and without the first patch (ompi-server
 as orted) but it doesn't change the behavior.

 I've attached gdb to my server, this is bt:
 #0  0xe410 in __kernel_vsyscall ()
 #1  0x00637afc in sched_yield () from /lib/libc.so.6
 #2  0xf7e8ce31 in opal_progress () at 
 ../../opal/runtime/opal_progress.c:220
 #3  0xf7f60ad4 in opal_condition_wait (c=0xf7fd7dc0, m=0xf7fd7e00) at
 ../../opal/threads/condition.h:99
 #4  0xf7f60dee in ompi_request_default_wait_all (count=2,
 requests=0xff8d7754, statuses=0x0) at
 ../../ompi/request/req_wait.c:262
 #5  0xf7d3e221 in mca_coll_inter_allgatherv_inter (sbuf=0xff8d7824,
 scount=1, sdtype=0x8049200, rbuf=0xff8d77e0, rcounts=0x9783df8,
 disps=0x9755520, rdtype=0x8049200, comm=0x978c2a8, module=0x9794b08)
at ../../../../../ompi/mca/coll/inter/coll_inter_allgatherv.c:127
 #6  0xf7f4c615 in ompi_comm_determine_first (intercomm=0x978c2a8,
 high=0) at ../../ompi/communicator/comm.c:1199
 #7  0xf7f8d1d9 in PMPI_Intercomm_merge (intercomm=0x978c2a8, high=0,
 newcomm=0xff8d78c0) at pintercomm_merge.c:84
 #8  0x0804893c in main (argc=Cannot access memory at address 0xf
 ) at serv

Re: [OMPI users] MPI_Allreduce on local machine

[Jeff Squyres]

I don't have the intel compilers on my Mac, but I'm unable to replicate this 
issue on Linux with the intel compilers v11.0.

Can you get a corefile to see a backtrace where it died in Open MPI's allreduce?

How exactly did you configure your Open MPI, and how exactly did you compile / 
run your sample application?


On Jul 27, 2010, at 10:35 PM, Hugo Gagnon wrote:

> I did and it runs now, but the result is wrong: outside is still 1.d0,
> 2.d0, 3.d0, 4.d0, 5.d0
> How can I make sure to compile OpenMPI so that datatypes such as
> mpi_double_precision behave as they "should"?
> Are there flags during the OpenMPI building process or something?
> Thanks,
> --
>   Hugo Gagnon
> 
> 
> On Tue, 27 Jul 2010 09:06 -0700, "David Zhang" 
> wrote:
> > Try mpi_real8 for the type in allreduce
> >
> > On 7/26/10, Hugo Gagnon  wrote:
> > > Hello,
> > >
> > > When I compile and run this code snippet:
> > >
> > >   1 program test
> > >   2
> > >   3 use mpi
> > >   4
> > >   5 implicit none
> > >   6
> > >   7 integer :: ierr, nproc, myrank
> > >   8 integer, parameter :: dp = kind(1.d0)
> > >   9 real(kind=dp) :: inside(5), outside(5)
> > >  10
> > >  11 call mpi_init(ierr)
> > >  12 call mpi_comm_size(mpi_comm_world, nproc, ierr)
> > >  13 call mpi_comm_rank(mpi_comm_world, myrank, ierr)
> > >  14
> > >  15 inside = (/ 1, 2, 3, 4, 5 /)
> > >  16 call mpi_allreduce(inside, outside, 5, mpi_double_precision,
> > >  mpi_sum, mpi_comm_world, ierr)
> > >  17
> > >  18 print*, myrank, inside
> > >  19 print*, outside
> > >  20
> > >  21 call mpi_finalize(ierr)
> > >  22
> > >  23 end program test
> > >
> > > I get the following error, with say 2 processors:
> > >
> > > forrtl: severe (174): SIGSEGV, segmentation fault occurred
> > > Image  PCRoutineLine
> > > Source
> > > libmpi.0.dylib 0001001BB4B7  Unknown   Unknown
> > > Unknown
> > > libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown
> > > Unknown
> > > a.out  00010CE2  _MAIN__16
> > > test.f90
> > > a.out  00010BDC  Unknown   Unknown
> > > Unknown
> > > a.out  00010B74  Unknown   Unknown
> > > Unknown
> > > forrtl: severe (174): SIGSEGV, segmentation fault occurred
> > > Image  PCRoutineLine
> > > Source
> > > libmpi.0.dylib 0001001BB4B7  Unknown   Unknown
> > > Unknown
> > > libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown
> > > Unknown
> > > a.out  00010CE2  _MAIN__16
> > > test.f90
> > > a.out  00010BDC  Unknown   Unknown
> > > Unknown
> > > a.out  00010B74  Unknown   Unknown
> > > Unknown
> > >
> > > on my iMac having compiled OpenMPI with ifort according to:
> > > http://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-open-mpi-with-the-intel-compilers/
> > >
> > > Note that the above code snippet runs fine on my school parallel cluster
> > > where ifort+intelmpi is installed.
> > > Is there something special about OpenMPI's MPI_Allreduce function call
> > > that I should be aware of?
> > >
> > > Thanks,
> > > --
> > >   Hugo Gagnon
> > >
> > > ___
> > > users mailing list
> > > us...@open-mpi.org
> > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >
> >
> > --
> > Sent from my mobile device
> >
> > David Zhang
> > University of California, San Diego
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >
> --
>   Hugo Gagnon
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] OpenMPI providing rank?

[Yves Caniou]

Le Wednesday 28 July 2010 15:05:28, vous avez écrit :
> I am confused. I thought all you wanted to do is report out the binding of
> the process - yes? Are you trying to set the affinity bindings yourself?
>
> If the latter, then your script doesn't do anything that mpirun wouldn't
> do, and doesn't do it as well. You would be far better off just adding
> --bind-to-core to the mpirun cmd line.

"mpirun -h" says that it is the default, so there is not even something to do?
I don't even have to add "--mca mpi_paffinity_alone 1" ?

.Yves.

> On Jul 28, 2010, at 6:37 AM, Yves Caniou wrote:
> > Le Wednesday 28 July 2010 11:34:13 Ralph Castain, vous avez écrit :
> >> On Jul 27, 2010, at 11:18 PM, Yves Caniou wrote:
> >>> Le Wednesday 28 July 2010 06:03:21 Nysal Jan, vous avez écrit :
>  OMPI_COMM_WORLD_RANK can be used to get the MPI rank. For other
>  environment variables -
>  http://www.open-mpi.org/faq/?category=running#mpi-environmental-variab
> le s
> >>>
> >>> Are processes affected to nodes sequentially, so that I can get the
> >>> NODE number from $OMPI_COMM_WORLD_RANK modulo the number of proc per
> >>> node?
> >>
> >> By default, yes. However, you can select alternative mapping methods.
> >>
> >> Or...you could just use the mpirun cmd line option to report the binding
> >> of each process as it is started :-)
> >>
> >> Do "mpirun -h" to see all the options. The one you want is
> >> --report-bindings
> >
> > It reports to stderr, so the $OMPI_COMM_WORLD_RANK modulo the number of
> > proc per nodes seems more appropriate for what I need, right?
> >
> > So is the following valid to put memory affinity?
> >
> > script.sh:
> >  MYRANK=$OMPI_COMM_WORLD_RANK
> >  MYVAL=$(expr $MYRANK / 4)
> >  NODE=$(expr $MYVAL % 4)
> >  numactl --cpunodebind=$NODE --membind=$NODE $@
> >
> > mpiexec ./script.sh -n 128 myappli myparam
> >
>  For processor affinity see this FAQ entry -
>  http://www.open-mpi.org/faq/?category=all#using-paffinity
> >>>
> >>> Thank you, but that's where I had the information that I put in my
> >>> previous mail, so it doesn't answer to my question.
> >>
> >> Memory affinity is taken care of under-the-covers when paffinity is
> >> active. No other options are required.
> >
> > Which is better: using this option, or the cmd line with numactl (if it
> > works)? What is the difference?
> >
> > Tkx.
> >
> > .Yves.
> >
> >>> .Yves.
> >>>
>  --Nysal
> 
>  On Wed, Jul 28, 2010 at 9:04 AM, Yves Caniou
> >
> > wrote:
> > Hi,
> >
> > I have some performance issue on a parallel machine composed of nodes
> > of 16 procs each. The application is launched on multiple of 16 procs
> > for given numbers of nodes.
> > I was told by people using MX MPI with this machine to attach a
> > script to mpiexec, which 'numactl' things, in order to make the
> > execution performance stable.
> >
> > Looking on the faq (the oldest one is for OpenMPI v1.3?), I saw that
> > maybe the
> > solution would be for me to use the --mca mpi_paffinity_alone 1
> >
> > Is that correct? -- BTW, I have both memory and processor affinity:
> >> ompi_info | grep affinity
> >
> > MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4.2)
> > MCA maffinity: first_use (MCA v2.0, API v2.0, Component
> > v1.4.2) MCA maffinity: libnuma (MCA v2.0, API v2.0, Component v1.4.2)
> > Does it handle memory too, or do I have to use another option like
> > --mca mpi_maffinity 1?
> >
> > Still, I would like to test the numactl solution. Does OpenMPI
> > provide an equivalent to $MXMPI_ID which gives at least gives the
> > NODE on which a process is launched by OpenMPI, so that I can adapt
> > the script that was given
> > to me?
> >
> > Tkx.
> >
> > .Yves.
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> >>>
> >>> ___
> >>> users mailing list
> >>> us...@open-mpi.org
> >>> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Allreduce on local machine

[Hugo Gagnon]

And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
MPI_DOUBLE_PRECISION and the result was 8. So I changed my program to:

  1 program test
  2 
  3 use mpi
  4 
  5 implicit none
  6 
  7 integer :: ierr, nproc, myrank
  8 !integer, parameter :: dp = kind(1.d0)
  9 real(kind=8) :: inside(5), outside(5)
 10 
 11 call mpi_init(ierr)
 12 call mpi_comm_size(mpi_comm_world, nproc, ierr)
 13 call mpi_comm_rank(mpi_comm_world, myrank, ierr)
 14 
 15 inside = (/ 1., 2., 3., 4., 5. /)
 16 call mpi_allreduce(inside, outside, 5, mpi_real, mpi_sum,
 mpi_comm_world, ierr)
 17 
 18 if (myrank == 0) then
 19 print*, outside
 20 end if
 21 
 22 call mpi_finalize(ierr)
 23 
 24 end program test

but I still get a SIGSEGV fault:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image  PCRoutineLine   
Source 
libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
Unknown
libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
Unknown
a.out  00010D87  _MAIN__16 
test.f90
a.out  00010C9C  Unknown   Unknown 
Unknown
a.out  00010C34  Unknown   Unknown 
Unknown
forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image  PCRoutineLine   
Source 
libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
Unknown
libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
Unknown
a.out  00010D87  _MAIN__16 
test.f90
a.out  00010C9C  Unknown   Unknown 
Unknown
a.out  00010C34  Unknown   Unknown 
Unknown

What is wrong now?
-- 
  Hugo Gagnon


On Wed, 28 Jul 2010 07:56 -0400, "Jeff Squyres" 
wrote:
> On Jul 27, 2010, at 4:19 PM, Gus Correa wrote:
> 
> > Is there a simple way to check the number of bytes associated to each
> > MPI basic type of OpenMPI on a specific machine (or machine+compiler)?
> > 
> > Something that would come out easily, say, from ompi_info?
> 
> Not via ompi_info, but the MPI function MPI_GET_EXTENT will tell you the
> datatype's size.
> 
> -
> [4:54] svbu-mpi:~/mpi % cat extent.f90
>   program main
>   use mpi
>   implicit none
>   integer ierr, ext
>   
>   call MPI_INIT(ierr)
>   call MPI_TYPE_EXTENT(MPI_DOUBLE_PRECISION, ext, ierr)
>   print *, 'Type extent of DOUBLE_PREC is', ext
>   call MPI_FINALIZE(ierr)
>   
>   end
> [4:54] svbu-mpi:~/mpi % mpif90 extent.f90 -o extent -g
> [4:54] svbu-mpi:~/mpi % ./extent
>  Type extent of DOUBLE_PREC is   8
> [4:54] svbu-mpi:~/mpi % 
> -
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
-- 
  Hugo Gagnon

Re: [OMPI users] MPI_Allreduce on local machine

[Hugo Gagnon]

I installed with:

./configure --prefix=/opt/openmpi CC=icc CXX=icpc F77=ifort FC=ifort
make all install

I would gladly give you a corefile but I have no idea on to produce one,
I'm just an end user...
-- 
  Hugo Gagnon


On Wed, 28 Jul 2010 08:57 -0400, "Jeff Squyres" 
wrote:
> I don't have the intel compilers on my Mac, but I'm unable to replicate
> this issue on Linux with the intel compilers v11.0.
> 
> Can you get a corefile to see a backtrace where it died in Open MPI's
> allreduce?
> 
> How exactly did you configure your Open MPI, and how exactly did you
> compile / run your sample application?
> 
> 
> On Jul 27, 2010, at 10:35 PM, Hugo Gagnon wrote:
> 
> > I did and it runs now, but the result is wrong: outside is still 1.d0,
> > 2.d0, 3.d0, 4.d0, 5.d0
> > How can I make sure to compile OpenMPI so that datatypes such as
> > mpi_double_precision behave as they "should"?
> > Are there flags during the OpenMPI building process or something?
> > Thanks,
> > --
> >   Hugo Gagnon
> > 
> > 
> > On Tue, 27 Jul 2010 09:06 -0700, "David Zhang" 
> > wrote:
> > > Try mpi_real8 for the type in allreduce
> > >
> > > On 7/26/10, Hugo Gagnon  wrote:
> > > > Hello,
> > > >
> > > > When I compile and run this code snippet:
> > > >
> > > >   1 program test
> > > >   2
> > > >   3 use mpi
> > > >   4
> > > >   5 implicit none
> > > >   6
> > > >   7 integer :: ierr, nproc, myrank
> > > >   8 integer, parameter :: dp = kind(1.d0)
> > > >   9 real(kind=dp) :: inside(5), outside(5)
> > > >  10
> > > >  11 call mpi_init(ierr)
> > > >  12 call mpi_comm_size(mpi_comm_world, nproc, ierr)
> > > >  13 call mpi_comm_rank(mpi_comm_world, myrank, ierr)
> > > >  14
> > > >  15 inside = (/ 1, 2, 3, 4, 5 /)
> > > >  16 call mpi_allreduce(inside, outside, 5, mpi_double_precision,
> > > >  mpi_sum, mpi_comm_world, ierr)
> > > >  17
> > > >  18 print*, myrank, inside
> > > >  19 print*, outside
> > > >  20
> > > >  21 call mpi_finalize(ierr)
> > > >  22
> > > >  23 end program test
> > > >
> > > > I get the following error, with say 2 processors:
> > > >
> > > > forrtl: severe (174): SIGSEGV, segmentation fault occurred
> > > > Image  PCRoutineLine
> > > > Source
> > > > libmpi.0.dylib 0001001BB4B7  Unknown   Unknown
> > > > Unknown
> > > > libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown
> > > > Unknown
> > > > a.out  00010CE2  _MAIN__16
> > > > test.f90
> > > > a.out  00010BDC  Unknown   Unknown
> > > > Unknown
> > > > a.out  00010B74  Unknown   Unknown
> > > > Unknown
> > > > forrtl: severe (174): SIGSEGV, segmentation fault occurred
> > > > Image  PCRoutineLine
> > > > Source
> > > > libmpi.0.dylib 0001001BB4B7  Unknown   Unknown
> > > > Unknown
> > > > libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown
> > > > Unknown
> > > > a.out  00010CE2  _MAIN__16
> > > > test.f90
> > > > a.out  00010BDC  Unknown   Unknown
> > > > Unknown
> > > > a.out  00010B74  Unknown   Unknown
> > > > Unknown
> > > >
> > > > on my iMac having compiled OpenMPI with ifort according to:
> > > > http://software.intel.com/en-us/articles/performance-tools-for-software-developers-building-open-mpi-with-the-intel-compilers/
> > > >
> > > > Note that the above code snippet runs fine on my school parallel cluster
> > > > where ifort+intelmpi is installed.
> > > > Is there something special about OpenMPI's MPI_Allreduce function call
> > > > that I should be aware of?
> > > >
> > > > Thanks,
> > > > --
> > > >   Hugo Gagnon
> > > >
> > > > ___
> > > > users mailing list
> > > > us...@open-mpi.org
> > > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > >
> > >
> > > --
> > > Sent from my mobile device
> > >
> > > David Zhang
> > > University of California, San Diego
> > > ___
> > > users mailing list
> > > us...@open-mpi.org
> > > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > >
> > --
> >   Hugo Gagnon
> > 
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > 
> 
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
-- 
  Hugo Gagnon

Re: [OMPI users] MPI_Allreduce on local machine

[Hugo Gagnon]

I mean to write:
call mpi_allreduce(inside, outside, 5,mpi_real, mpi_double_precision,
mpi_comm_world, ierr)
-- 
  Hugo Gagnon


On Wed, 28 Jul 2010 09:33 -0400, "Hugo Gagnon"
 wrote:
> And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
> And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
> MPI_DOUBLE_PRECISION and the result was 8. So I changed my program to:
> 
>   1 program test
>   2 
>   3 use mpi
>   4 
>   5 implicit none
>   6 
>   7 integer :: ierr, nproc, myrank
>   8 !integer, parameter :: dp = kind(1.d0)
>   9 real(kind=8) :: inside(5), outside(5)
>  10 
>  11 call mpi_init(ierr)
>  12 call mpi_comm_size(mpi_comm_world, nproc, ierr)
>  13 call mpi_comm_rank(mpi_comm_world, myrank, ierr)
>  14 
>  15 inside = (/ 1., 2., 3., 4., 5. /)
>  16 call mpi_allreduce(inside, outside, 5, mpi_real, mpi_sum,
>  mpi_comm_world, ierr)
>  17 
>  18 if (myrank == 0) then
>  19 print*, outside
>  20 end if
>  21 
>  22 call mpi_finalize(ierr)
>  23 
>  24 end program test
> 
> but I still get a SIGSEGV fault:
> 
> forrtl: severe (174): SIGSEGV, segmentation fault occurred
> Image  PCRoutineLine   
> Source 
> libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
> Unknown
> libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
> Unknown
> a.out  00010D87  _MAIN__16 
> test.f90
> a.out  00010C9C  Unknown   Unknown 
> Unknown
> a.out  00010C34  Unknown   Unknown 
> Unknown
> forrtl: severe (174): SIGSEGV, segmentation fault occurred
> Image  PCRoutineLine   
> Source 
> libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
> Unknown
> libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
> Unknown
> a.out  00010D87  _MAIN__16 
> test.f90
> a.out  00010C9C  Unknown   Unknown 
> Unknown
> a.out  00010C34  Unknown   Unknown 
> Unknown
> 
> What is wrong now?
> -- 
>   Hugo Gagnon
> 
> 
> On Wed, 28 Jul 2010 07:56 -0400, "Jeff Squyres" 
> wrote:
> > On Jul 27, 2010, at 4:19 PM, Gus Correa wrote:
> > 
> > > Is there a simple way to check the number of bytes associated to each
> > > MPI basic type of OpenMPI on a specific machine (or machine+compiler)?
> > > 
> > > Something that would come out easily, say, from ompi_info?
> > 
> > Not via ompi_info, but the MPI function MPI_GET_EXTENT will tell you the
> > datatype's size.
> > 
> > -
> > [4:54] svbu-mpi:~/mpi % cat extent.f90
> >   program main
> >   use mpi
> >   implicit none
> >   integer ierr, ext
> >   
> >   call MPI_INIT(ierr)
> >   call MPI_TYPE_EXTENT(MPI_DOUBLE_PRECISION, ext, ierr)
> >   print *, 'Type extent of DOUBLE_PREC is', ext
> >   call MPI_FINALIZE(ierr)
> >   
> >   end
> > [4:54] svbu-mpi:~/mpi % mpif90 extent.f90 -o extent -g
> > [4:54] svbu-mpi:~/mpi % ./extent
> >  Type extent of DOUBLE_PREC is   8
> > [4:54] svbu-mpi:~/mpi % 
> > -
> > 
> > -- 
> > Jeff Squyres
> > jsquy...@cisco.com
> > For corporate legal information go to:
> > http://www.cisco.com/web/about/doing_business/legal/cri/
> > 
> > 
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
> > 
> -- 
>   Hugo Gagnon
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
-- 
  Hugo Gagnon

Re: [OMPI users] openMPI shared with NFS, but says different version

[Cristobal Navarro]

yes,

somehow after the second install, the installlation is consistent.

im only running into an issue, might be mpi im not sure.
these nodes, each one have 8 phisical procesors (2xIntel Xeon quad core),
and 16 virtual ones, btw i have ubuntu server 64bit 10.04 instaled on these
nodes.

the problem seems to be whenever y try to use over 8 proceses (make use of
the virtual ones), i get a horrible error saying about a kernel error and a
certain cpu that crashed, the error hags there for about a minute, then it
switches to another cpu and shows the same error. i have no other option to
press power off button.

ill try to copy the error, and post it.



On Wed, Jul 28, 2010 at 7:39 AM, Jeff Squyres  wrote:

> This issue is usually caused by installing one version of Open MPI over an
> older version:
>
>http://www.open-mpi.org/faq/?category=building#install-overwrite
>
>
> On Jul 27, 2010, at 10:35 PM, Cristobal Navarro wrote:
>
> >
> > On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa 
> wrote:
> > Hi Cristobal
> >
> > Does it run only on the head node alone?
> > (Fuego? Agua? Acatenango?)
> > Try to put only the head node on the hostfile and execute with mpiexec.
> > --> i will try only with the head node, and post results back
> > This may help sort out what is going on.
> > Hopefully it will run on the head node.
> >
> > Also, do you have Infinband connecting the nodes?
> > The error messages refer to the openib btl (i.e. Infiniband),
> > and complains of
> >
> > no we are just using normal network 100MBit/s , since i am just testing
> yet.
> >
> > "perhaps a missing symbol, or compiled for a different
> > version of Open MPI?".
> > It sounds as a mixup of versions/builds.
> >
> > --> i agree, somewhere there must be the remains of the older version
> >
> > Did you configure/build OpenMPI from source, or did you install
> > it with apt-get?
> > It may be easier/less confusing to install from source.
> > If you did, what configure options did you use?
> >
> > -->i installed from source,
> > ./configure --prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid
> --disable--static
> >
> > Also, as for the OpenMPI runtime environment,
> > it is not enough to set it on
> > the command line, because it will be effective only on the head node.
> > You need to either add them to the PATH and LD_LIBRARY_PATH
> > on your .bashrc/.cshrc files (assuming these files and your home
> directory are *also* shared with the nodes via NFS),
> > or use the --prefix option of mpiexec to point to the OpenMPI main
> directory.
> >
> > yes, all nodes have their PATH and LD_LIBRARY_PATH set up properly inside
> the login scripts ( .bashrc in my case  )
> >
> > Needless to say, you need to check and ensure that the OpenMPI directory
> (and maybe your home directory, and your work directory) is (are)
> > really mounted on the nodes.
> >
> > --> yes, doublechecked that they are
> >
> > I hope this helps,
> >
> > --> thanks really!
> >
> > Gus Correa
> >
> > Update: i just reinstalled openMPI, with the same parameters, and it
> seems that the problem has gone, i couldnt test entirely but when i get back
> to lab ill confirm.
> >
> > best regards!
> > Cristobal
> >
> > ___
> > users mailing list
> > us...@open-mpi.org
> > http://www.open-mpi.org/mailman/listinfo.cgi/users
>
>
> --
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] openMPI shared with NFS, but says different version

[Gus Correa]

Hi Cristobal

In case you are not using full path name for mpiexec/mpirun,
what does "which mpirun" say?

Often times this is a source of confusion, old versions may
be first on the PATH.

Gus

Cristobal Navarro wrote:


On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa > wrote:


Hi Cristobal

Does it run only on the head node alone?
(Fuego? Agua? Acatenango?)
Try to put only the head node on the hostfile and execute with mpiexec.

--> i will try only with the head node, and post results back 


This may help sort out what is going on.
Hopefully it will run on the head node.

Also, do you have Infinband connecting the nodes?
The error messages refer to the openib btl (i.e. Infiniband),
and complains of


no we are just using normal network 100MBit/s , since i am just testing yet.


"perhaps a missing symbol, or compiled for a different
version of Open MPI?".
It sounds as a mixup of versions/builds.


--> i agree, somewhere there must be the remains of the older version 



Did you configure/build OpenMPI from source, or did you install
it with apt-get?
It may be easier/less confusing to install from source.
If you did, what configure options did you use?


-->i installed from source, 
./configure --prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid 
--disable--static 



Also, as for the OpenMPI runtime environment,
it is not enough to set it on
the command line, because it will be effective only on the head node.
You need to either add them to the PATH and LD_LIBRARY_PATH
on your .bashrc/.cshrc files (assuming these files and your home
directory are *also* shared with the nodes via NFS),
or use the --prefix option of mpiexec to point to the OpenMPI main
directory.


yes, all nodes have their PATH and LD_LIBRARY_PATH set up properly 
inside the login scripts ( .bashrc in my case  ) 



Needless to say, you need to check and ensure that the OpenMPI
directory (and maybe your home directory, and your work directory)
is (are)
really mounted on the nodes.


--> yes, doublechecked that they are 



I hope this helps,


--> thanks really! 



Gus Correa

Update: i just reinstalled openMPI, with the same parameters, and it
seems that the problem has gone, i couldnt test entirely but when i
get back to lab ill confirm. 



best regards! 
Cristobal





___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI_Allreduce on local machine

[Gus Correa]

Hugo Gagnon wrote:

Hi Gus,
Ompi_info --all lists its info regarding fortran right after C. In my
case:
  Fort real size: 4
 Fort real4 size: 4
 Fort real8 size: 8
Fort real16 size: 16
  Fort dbl prec size: 4
Does it make any sense to you?


Hi Hugo

No, dbl prec size 4 sounds weird, should be 8, I suppose,
same as real8, right?

It doesn't make sense, but that's what I have (now that you told me
that "dbl" , not "double", is the string to search for):

$  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4

Is this a bug in OpenMPI perhaps?

I didn't come across to this problem, most likely because
the codes here don't use "double precision" but real*8 or similar.

Also make sure you are picking the right ompi_info, mpif90/f77, mpiexec.
Often times old versions and tangled PATH make things very confusing.

Gus Correa

Re: [OMPI users] MPI_Allreduce on local machine

[Jeff Squyres]

On Jul 28, 2010, at 11:19 AM, Gus Correa wrote:

> > Ompi_info --all lists its info regarding fortran right after C. In my

Ummm right... I should know that.  I wrote ompi_info, after all.  :-)  I 
ran "ompi_info -all | grep -i fortran" and didn't see the fortran info, and I 
forgot that I put that stuff in there.  Oops!  :-)

> > case:
> >   Fort real size: 4
> >  Fort real4 size: 4
> >  Fort real8 size: 8
> > Fort real16 size: 16
> >   Fort dbl prec size: 4

That does seems weird.  

> No, dbl prec size 4 sounds weird, should be 8, I suppose,
> same as real8, right?
> 
> It doesn't make sense, but that's what I have (now that you told me
> that "dbl" , not "double", is the string to search for):
> 
> $  Fort dbl prec size: 4
>Fort dbl cplx size: 4
>   Fort dbl prec align: 4
>   Fort dbl cplx align: 4
> 
> Is this a bug in OpenMPI perhaps?

Getting sizeof and alignment of Fortran variable types is very problematic.  
Can you send the stdout/stderr of configure and the config.log?

http://www.open-mpi.org/community/help/

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_Allreduce on local machine

[Gus Correa]

Hi Hugo, Jeff, list

Hugo: I think David Zhang's suggestion was to use
MPI_REAL8 not MPI_REAL, instead of MPI_DOUBLE_PRECISION in your
MPI_Allreduce call.

Still, to me it looks like OpenMPI is making double precision 4-byte 
long, which shorter than I expected it be (8 bytes),

at least when looking at the output of ompi_info --all.

I always get get a size 4 for dbl prec in my x86_64 machine,
from ompi_info --all.
I confirmed this in six builds of OpenMPI 1.4.2:  gcc+gfortran,
gcc+pgf90, gcc+ifort, icc+ifort, pgcc+pgf90, and opencc+openf95.
Although the output of ompi_info never says this is actually the size
of MPI_DOUBLE_PRECISION, just of "dbl prec", which is a bit ambiguous.

FWIW, I include the output below.  Note that alignment for gcc+ifort
is 1, all others are 4.

Jeff:  Is this correct?

Thanks,
Gus Correa


$ openmpi/1.4.2/open64-4.2.3-0/bin/ompi_info --all | grep -i dbl
  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4
$ openmpi/1.4.2/gnu-4.1.2/bin/ompi_info --all | grep -i dbl
  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4
$ openmpi/1.4.2/gnu-4.1.2-intel-10.1.017/bin/ompi_info --all | grep -i dbl
  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 1
 Fort dbl cplx align: 1
$ openmpi/1.4.2/gnu-4.1.2-pgi-8.0-4/bin/ompi_info --all | grep -i dbl
  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4
$ openmpi/1.4.2/pgi-8.0-4/bin/ompi_info --all | grep -i dbl
  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4

Hugo Gagnon wrote:

And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
And how do I know how big my data buffer is? I ran MPI_TYPE_EXTENT of
MPI_DOUBLE_PRECISION and the result was 8. So I changed my program to:

  1 program test
  2 
  3 use mpi
  4 
  5 implicit none
  6 
  7 integer :: ierr, nproc, myrank

  8 !integer, parameter :: dp = kind(1.d0)
  9 real(kind=8) :: inside(5), outside(5)
 10 
 11 call mpi_init(ierr)

 12 call mpi_comm_size(mpi_comm_world, nproc, ierr)
 13 call mpi_comm_rank(mpi_comm_world, myrank, ierr)
 14 
 15 inside = (/ 1., 2., 3., 4., 5. /)

 16 call mpi_allreduce(inside, outside, 5, mpi_real, mpi_sum,
 mpi_comm_world, ierr)
 17 
 18 if (myrank == 0) then

 19 print*, outside
 20 end if
 21 
 22 call mpi_finalize(ierr)
 23 
 24 end program test


but I still get a SIGSEGV fault:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image  PCRoutineLine   
Source 
libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
Unknown
libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
Unknown
a.out  00010D87  _MAIN__16 
test.f90
a.out  00010C9C  Unknown   Unknown 
Unknown
a.out  00010C34  Unknown   Unknown 
Unknown

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image  PCRoutineLine   
Source 
libmpi.0.dylib 0001001BB4B7  Unknown   Unknown 
Unknown
libmpi_f77.0.dyli  0001000AF046  Unknown   Unknown 
Unknown
a.out  00010D87  _MAIN__16 
test.f90
a.out  00010C9C  Unknown   Unknown 
Unknown
a.out  00010C34  Unknown   Unknown 
Unknown


What is wrong now?

Re: [OMPI users] MPI_Allreduce on local machine

[Gus Correa]

Hi Jeff

I surely can send you the logs, but they're big.
Off the list perhaps?

Thanks,
Gus

Jeff Squyres wrote:

On Jul 28, 2010, at 11:19 AM, Gus Correa wrote:


Ompi_info --all lists its info regarding fortran right after C. In my


Ummm right... I should know that.  I wrote ompi_info, after all.  :-)  I ran 
"ompi_info -all | grep -i fortran" and didn't see the fortran info, and I 
forgot that I put that stuff in there.  Oops!  :-)


case:
  Fort real size: 4
 Fort real4 size: 4
 Fort real8 size: 8
Fort real16 size: 16
  Fort dbl prec size: 4


That does seems weird.  


No, dbl prec size 4 sounds weird, should be 8, I suppose,
same as real8, right?

It doesn't make sense, but that's what I have (now that you told me
that "dbl" , not "double", is the string to search for):

$  Fort dbl prec size: 4
   Fort dbl cplx size: 4
  Fort dbl prec align: 4
  Fort dbl cplx align: 4

Is this a bug in OpenMPI perhaps?


Getting sizeof and alignment of Fortran variable types is very problematic.  
Can you send the stdout/stderr of configure and the config.log?

http://www.open-mpi.org/community/help/

Re: [OMPI users] MPI_Allreduce on local machine

[Jeff Squyres]

On Jul 28, 2010, at 11:55 AM, Gus Correa wrote:

> I surely can send you the logs, but they're big.
> Off the list perhaps?

If they're still big when compressed, sure, send them to me off list.

But I think I'd be more interested to see Hugo's logs.  :-)

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI_Allreduce on local machine

[Hugo Gagnon]

Here they are.
-- 
  Hugo Gagnon


On Wed, 28 Jul 2010 12:01 -0400, "Jeff Squyres" 
wrote:
> On Jul 28, 2010, at 11:55 AM, Gus Correa wrote:
> 
> > I surely can send you the logs, but they're big.
> > Off the list perhaps?
> 
> If they're still big when compressed, sure, send them to me off list.
> 
> But I think I'd be more interested to see Hugo's logs.  :-)
> 
> -- 
> Jeff Squyres
> jsquy...@cisco.com
> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
> 
-- 
  Hugo Gagnon



ompi-output.tar.bz2
Description: BZip2 compressed data

Re: [OMPI users] MPI_Allreduce on local machine

[Åke Sandgren]

On Wed, 2010-07-28 at 11:48 -0400, Gus Correa wrote:
> Hi Hugo, Jeff, list
> 
> Hugo: I think David Zhang's suggestion was to use
> MPI_REAL8 not MPI_REAL, instead of MPI_DOUBLE_PRECISION in your
> MPI_Allreduce call.
> 
> Still, to me it looks like OpenMPI is making double precision 4-byte 
> long, which shorter than I expected it be (8 bytes),
> at least when looking at the output of ompi_info --all.
> 
> I always get get a size 4 for dbl prec in my x86_64 machine,
> from ompi_info --all.
> I confirmed this in six builds of OpenMPI 1.4.2:  gcc+gfortran,
> gcc+pgf90, gcc+ifort, icc+ifort, pgcc+pgf90, and opencc+openf95.
> Although the output of ompi_info never says this is actually the size
> of MPI_DOUBLE_PRECISION, just of "dbl prec", which is a bit ambiguous.
> 
> FWIW, I include the output below.  Note that alignment for gcc+ifort
> is 1, all others are 4.
> 
> Jeff:  Is this correct?

This is wrong, it should be 8 and alignement should be 8 even for intel.
And i also see exactly the same thing.

-- 
Ake Sandgren, HPC2N, Umea University, S-90187 Umea, Sweden
Internet: a...@hpc2n.umu.se   Phone: +46 90 7866134 Fax: +46 90 7866126
Mobile: +46 70 7716134 WWW: http://www.hpc2n.umu.se

[OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[christophe petit]

hello,

i have a problem concerning the execution of a f90 program (explicitPar)
compiled with openmpi-1.4.2. I get nearly the same error on my debian
desktop ( AMD Phenom(tm) 9550 Quad-Core Processor) and my mac pro i7 laptop
:

on mac pro i7 :

$ mpiexec -np 2 explicitPar
[macbook-pro-de-fab.livebox.home:48805] *** An error occurred in
MPI_Cart_shift
[macbook-pro-de-fab.livebox.home:48805] *** on communicator MPI_COMM_WORLD
[macbook-pro-de-fab.livebox.home:48805] *** MPI_ERR_COMM: invalid
communicator
[macbook-pro-de-fab.livebox.home:48805] *** MPI_ERRORS_ARE_FATAL (your MPI
job will now abort)
--
mpiexec has exited due to process rank 1 with PID 48805 on
node macbook-pro-de-fab.livebox.home exiting without calling "finalize".
This may
have caused other processes in the application to be
terminated by signals sent by mpiexec (as reported here).

---

on my debian desktop :

mpirun -np 2 explicitPar
[pablo:11665] *** An error occurred in MPI_Cart_shift
[pablo:11665] *** on communicator MPI_COMM_WORLD
[pablo:11665] *** MPI_ERR_COMM: invalid communicator
[pablo:11665] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
--
mpirun has exited due to process rank 1 with PID 11665 on
node pablo exiting without calling "finalize". This may
have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).
--


I have installed openmpi-1.4.2 with the following options :

./configure --prefix=/usr/local/openmpi --enable-mpi-f77 --enable-mpi-f90

with exported variables on bash shell : FC=gfortran F90=gfortran
F77=gfortran CC=gcc CXX=g++

The  installation has been completed, the program compiles fine but i don't
understand what's wrong. I note that with a single processor ("mpirun -np 1
explicitPar"), execution works fine.

My debian desktop is a quad-core, so, theoretically, i can put up to 4 for
"np" parameter.
On my mac pro i7, i don't know how processors are there, but the "htop"
command makes appear 4 cores too.

Anyone has a solution ?

Regards.

Re: [OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[Ralph Castain]

First thing is to ensure you are getting the version of OMPI that you expect. 
Both the Mac and Debian come with their own pre-installed versions, so you have 
to ensure that PATH and LD_LIBRARY_PATH are correctly pointing to the version 
you installed and compiled against.


On Jul 28, 2010, at 10:28 AM, christophe petit wrote:

> hello,
> 
> i have a problem concerning the execution of a f90 program (explicitPar) 
> compiled with openmpi-1.4.2. I get nearly the same error on my debian desktop 
> ( AMD Phenom(tm) 9550 Quad-Core Processor) and my mac pro i7 laptop :
> 
> on mac pro i7 :
> 
> $ mpiexec -np 2 explicitPar
> [macbook-pro-de-fab.livebox.home:48805] *** An error occurred in 
> MPI_Cart_shift
> [macbook-pro-de-fab.livebox.home:48805] *** on communicator MPI_COMM_WORLD
> [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERR_COMM: invalid communicator
> [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERRORS_ARE_FATAL (your MPI 
> job will now abort)
> --
> mpiexec has exited due to process rank 1 with PID 48805 on
> node macbook-pro-de-fab.livebox.home exiting without calling "finalize". This 
> may
> have caused other processes in the application to be
> terminated by signals sent by mpiexec (as reported here).
> 
> ---
> 
> on my debian desktop :
> 
> mpirun -np 2 explicitPar
> [pablo:11665] *** An error occurred in MPI_Cart_shift
> [pablo:11665] *** on communicator MPI_COMM_WORLD
> [pablo:11665] *** MPI_ERR_COMM: invalid communicator
> [pablo:11665] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> --
> mpirun has exited due to process rank 1 with PID 11665 on
> node pablo exiting without calling "finalize". This may
> have caused other processes in the application to be
> terminated by signals sent by mpirun (as reported here).
> --
> 
> 
> I have installed openmpi-1.4.2 with the following options :
> 
> ./configure --prefix=/usr/local/openmpi --enable-mpi-f77 --enable-mpi-f90
> 
> with exported variables on bash shell : FC=gfortran F90=gfortran F77=gfortran 
> CC=gcc CXX=g++
> 
> The  installation has been completed, the program compiles fine but i don't 
> understand what's wrong. I note that with a single processor ("mpirun -np 1 
> explicitPar"), execution works fine.
> 
> My debian desktop is a quad-core, so, theoretically, i can put up to 4 for 
> "np" parameter.
> On my mac pro i7, i don't know how processors are there, but the "htop" 
> command makes appear 4 cores too.
> 
> Anyone has a solution ?
> 
> Regards.
> 
> 
> 
> 
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[Jeff Squyres]

According to the error message (especially since it's consistent across 2 
different platforms), it looks like you have an error in your application.  
Open MPI says that you're using an invalid communicator when calling 
MPI_Cart_shift.  "Invalid" probably means that it's not a Cartesian 
communicator.

You might want to double check the definition and requirements of the 
MPI_CART_SHIFT function (see the MPI_Cart_shift(3) man page).



On Jul 28, 2010, at 12:28 PM, christophe petit wrote:

> hello,
> 
> i have a problem concerning the execution of a f90 program (explicitPar) 
> compiled with openmpi-1.4.2. I get nearly the same error on my debian desktop 
> ( AMD Phenom(tm) 9550 Quad-Core Processor) and my mac pro i7 laptop :
> 
> on mac pro i7 :
> 
> $ mpiexec -np 2 explicitPar
> [macbook-pro-de-fab.livebox.home:48805] *** An error occurred in 
> MPI_Cart_shift
> [macbook-pro-de-fab.livebox.home:48805] *** on communicator MPI_COMM_WORLD
> [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERR_COMM: invalid communicator
> [macbook-pro-de-fab.livebox.home:48805] *** MPI_ERRORS_ARE_FATAL (your MPI 
> job will now abort)
> --
> mpiexec has exited due to process rank 1 with PID 48805 on
> node macbook-pro-de-fab.livebox.home exiting without calling "finalize". This 
> may
> have caused other processes in the application to be
> terminated by signals sent by mpiexec (as reported here).
> 
> ---
> 
> on my debian desktop :
> 
> mpirun -np 2 explicitPar
> [pablo:11665] *** An error occurred in MPI_Cart_shift
> [pablo:11665] *** on communicator MPI_COMM_WORLD
> [pablo:11665] *** MPI_ERR_COMM: invalid communicator
> [pablo:11665] *** MPI_ERRORS_ARE_FATAL (your MPI job will now abort)
> --
> mpirun has exited due to process rank 1 with PID 11665 on
> node pablo exiting without calling "finalize". This may
> have caused other processes in the application to be
> terminated by signals sent by mpirun (as reported here).
> --
> 
> 
> I have installed openmpi-1.4.2 with the following options :
> 
> ./configure --prefix=/usr/local/openmpi --enable-mpi-f77 --enable-mpi-f90
> 
> with exported variables on bash shell : FC=gfortran F90=gfortran F77=gfortran 
> CC=gcc CXX=g++
> 
> The  installation has been completed, the program compiles fine but i don't 
> understand what's wrong. I note that with a single processor ("mpirun -np 1 
> explicitPar"), execution works fine.
> 
> My debian desktop is a quad-core, so, theoretically, i can put up to 4 for 
> "np" parameter.
> On my mac pro i7, i don't know how processors are there, but the "htop" 
> command makes appear 4 cores too.
> 
> Anyone has a solution ?
> 
> Regards.
> 
> 
> 
> 
> 
> 
> 
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPIRUN Error on Mac pro i7 laptop and linux desktop

[christophe petit]

Thanks for your answers,

the execution of this parallel program works fine at my work, but we used
MPICH2. I thought this will run with OPEN-MPI too.

Here is the f90 source where MPI_CART_SHIFT is called :

  program heat
!**
!
!   This program solves the heat equation on the unit square [0,1]x[0,1]
!| du/dt - Delta(u) = 0
!|  u/gamma = cste
!   by implementing a explicit scheme.
!   The discretization is done using a 5 point finite difference scheme
!   and the domain is decomposed into sub-domains.
!   The PDE is discretized using a 5 point finite difference scheme
!   over a (x_dim+2)*(x_dim+2) grid including the end points
!   correspond to the boundary points that are stored.
!
!   The data on the whole domain are stored in
!   the following way :
!
!y
!   
!d  |  |
!i  |  |
!r  |  |
!e  |  |
!c  |  |
!t  |  |
!i  | x20  |
!o /\   |  |
!n  |   | x10  |
!   |   |  |
!   |   | x00  x01 x02 ... |
!   |   
!---> x direction  x(*,j)
!
!   The boundary conditions are stored in the following submatrices
!
!
!x(1:x_dim, 0)  ---> left   temperature
!x(1:x_dim, x_dim+1)---> right  temperature
!x(0, 1:x_dim)  ---> toptemperature
!x(x_dim+1, 1:x_dim)---> bottom temperature
!
!**
  implicit none
  include 'mpif.h'
! size of the discretization
  integer :: x_dim, nb_iter
  double precision, allocatable :: x(:,:),b(:,:),x0(:,:)
  double precision  :: dt, h, epsilon
  double precision  :: resLoc, result, t, tstart, tend
!
  integer :: i,j
  integer :: step, maxStep
  integer :: size_x, size_y, me, x_domains,y_domains
  integer :: iconf(5), size_x_glo
  double precision conf(2)
!
! MPI variables
  integer :: nproc, infompi, comm, comm2d, lda, ndims
  INTEGER, DIMENSION(2)  :: dims
  LOGICAL, DIMENSION(2)  :: periods
  LOGICAL, PARAMETER :: reorganisation = .false.
  integer :: row_type
  integer, parameter :: nbvi=4
  integer, parameter :: S=1, E=2, N=3, W=4
  integer, dimension(4) :: neighBor

!
  intrinsic abs
!
!
  call MPI_INIT(infompi)
  comm = MPI_COMM_WORLD
  call MPI_COMM_SIZE(comm,nproc,infompi)
  call MPI_COMM_RANK(comm,me,infompi)
!
!
  if (me.eq.0) then
  call readparam(iconf, conf)
  endif
  call MPI_BCAST(iconf,5,MPI_INTEGER,0,comm,infompi)
  call MPI_BCAST(conf,2,MPI_DOUBLE_PRECISION,0,comm,infompi)
!
  size_x= iconf(1)
  size_y= iconf(1)
  x_domains = iconf(3)
  y_domains = iconf(4)
  maxStep   = iconf(5)
  dt= conf(1)
  epsilon   = conf(2)
!
  size_x_glo = x_domains*size_x+2
  h  = 1.0d0/dble(size_x_glo)
  dt = 0.25*h*h
!
!
  lda = size_y+2
  allocate(x(0:size_y+1,0:size_x+1))
  allocate(x0(0:size_y+1,0:size_x+1))
  allocate(b(0:size_y+1,0:size_x+1))
!
! Create 2D cartesian grid
  periods(:) = .false.

  ndims = 2
  dims(1)=x_domains
  dims(2)=y_domains
  CALL MPI_CART_CREATE(MPI_COMM_WORLD, ndims, dims, periods, &
reorganisation,comm2d,infompi)
!
! Identify neighbors
!
  NeighBor(:) = MPI_PROC_NULL
! Left/West and right/Est neigbors
  CALL MPI_CART_SHIFT(comm2d,0,1,NeighBor(W),NeighBor(E),infompi)
! Bottom/South and Upper/North neigbors
  CALL MPI_CART_SHIFT(comm2d,1,1,NeighBor(S),NeighBor(N),infompi)
!
! Create row data type to coimmunicate with South and North neighbors
!
  CALL MPI_TYPE_VECTOR(size_x, 1, size_y+2, MPI_DOUBLE_PRECISION,
row_type,infompi)
  CALL MPI_TYPE_COMMIT(row_type, infompi)
!
! initialization
!
  call initvalues(x0, b, size_x+1, size_x )
!
! Update the boundaries
!
  call updateBound(x0,size_x,size_x, NeighBor, comm2d, row_type)

  step = 0
  t= 0.0
!
  tstart = MPI_Wtime()
! REPEAT
 10   continue
!
 step = step + 1
 t= t + dt
! perform one step of the explicit scheme
 call Explicit(x0,x,b, size_x+1, size_x, size_x, dt, h, resLoc)
! update the partial solution along the interface
 call updateBound(x0,size_x,size_x, NeighBor, comm2d, row_type)
! Check the distance between two iterates
 call MPI_ALLREDUCE(resLoc,result,1, MPI_DOUBLE_PRECISION,
MPI_SUM,comm,infompi)
 result= sqrt(result)
!
 if (me.eq.0) write(*,1002) t,result
!
   if 

Re: [OMPI users] Processes stuck after MPI_Waitall() in 1.4.1

[Brian Smith]

I've used the TCP btl and it works fine.  Its only with openib btl that
I have issues.  I have a set of nodes that uses qib and psm.  This mtl
works fine also.  I'll try adjusting rendezvous limit and message
settings as well as the collective algorithm options and see if that
helps.  

Many thanks for your assistance.  I'll report back with my findings.

-Brian

-- 
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB204
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu


On Wed, 2010-07-28 at 06:56 -0400, Terry Dontje wrote:
> Here are a couple other suggestions:
> 
> 1.  Have you tried your code with using the TCP btl just to make sure
> this might not be a general algorithm issue with the collective?
> 
> 2.  While using the openib btl you may want to try things with rdma
> turned off by using the following parameters to mpirun:
> -mca btl_openib_use_eager_rdma 0 -mca btl_openib_max_eager_rdma 0 -mca
> btl_openib_flags 1
> 
> 3.  While using the openib btl you may want to try things while
> bumping up the rendezvous limit to see  if eliminating rendezvous
> messages helps things (others on the list is there an easier way to do
> this?).  Set the following parameters raising the 8192 and 2048
> values:
> -mca btl_openib_receive_queues "P,8192" -mca btl_openib_max_send_size
> 8192 -mca btl_openib_eager_limit 8192 -mca btl_openib_rndv_eager_limit
> 2048
> 
> 4.  You mayb also want to try and see if the basic collective
> algorithm work instead of using the tuned, which is the default I
> believe, by setting  "-mca coll_basic_priority 100".  The idea here is
> to determine if the tuned collective itself is tickling the issue.
> 
> --td
> 
> Terry Dontje wrote: 
> > With this earlier failure do you know how many message may have been
> > transferred between the two processes?  Is there a way to narrow
> > this down to a small piece of code?  Do you have totalview or ddt at
> > your disposal?
> > 
> > --td
> > 
> > Brian Smith wrote: 
> > > Also, the application I'm having trouble with appears to work fine with
> > > MVAPICH2 1.4.1, if that is any help.
> > > 
> > > -Brian
> > > 
> > > On Tue, 2010-07-27 at 10:48 -0400, Terry Dontje wrote:
> > >   
> > > > Can you try a simple point-to-point program?
> > > > 
> > > > --td
> > > > 
> > > > Brian Smith wrote: 
> > > > 
> > > > > After running on two processors across two nodes, the problem occurs
> > > > > much earlier during execution:
> > > > > 
> > > > > (gdb) bt
> > > > > #0  opal_sys_timer_get_cycles ()
> > > > > at ../opal/include/opal/sys/amd64/timer.h:46
> > > > > #1  opal_timer_base_get_cycles ()
> > > > > at ../opal/mca/timer/linux/timer_linux.h:31
> > > > > #2  opal_progress () at runtime/opal_progress.c:181
> > > > > #3  0x2b4bc3c00215 in opal_condition_wait (count=2,
> > > > > requests=0x7fff33372480, statuses=0x7fff33372450)
> > > > > at ../opal/threads/condition.h:99
> > > > > #4  ompi_request_default_wait_all (count=2, requests=0x7fff33372480,
> > > > > statuses=0x7fff33372450) at request/req_wait.c:262
> > > > > #5  0x2b4bc3c915b7 in ompi_coll_tuned_sendrecv_actual
> > > > > (sendbuf=0x2aaad11dfaf0, scount=117692, 
> > > > > sdatatype=0x2b4bc3fa9ea0, dest=1, stag=-13, recvbuf= > > > > optimized
> > > > > out>, rcount=117692, 
> > > > > rdatatype=0x2b4bc3fa9ea0, source=1, rtag=-13, comm=0x12cd98c0,
> > > > > status=0x0) at coll_tuned_util.c:55
> > > > > #6  0x2b4bc3c982db in ompi_coll_tuned_sendrecv 
> > > > > (sbuf=0x2aaad10f9d10,
> > > > > scount=117692, sdtype=0x2b4bc3fa9ea0, 
> > > > > rbuf=0x2aaae104d010, rcount=117692, rdtype=0x2b4bc3fa9ea0,
> > > > > comm=0x12cd98c0, module=0x12cda340)
> > > > > at coll_tuned_util.h:60
> > > > > #7  ompi_coll_tuned_alltoall_intra_two_procs (sbuf=0x2aaad10f9d10,
> > > > > scount=117692, sdtype=0x2b4bc3fa9ea0, 
> > > > > rbuf=0x2aaae104d010, rcount=117692, rdtype=0x2b4bc3fa9ea0,
> > > > > comm=0x12cd98c0, module=0x12cda340)
> > > > > at coll_tuned_alltoall.c:432
> > > > > #8  0x2b4bc3c1b71f in PMPI_Alltoall (sendbuf=0x2aaad10f9d10,
> > > > > sendcount=117692, sendtype=0x2b4bc3fa9ea0, 
> > > > > recvbuf=0x2aaae104d010, recvcount=117692, recvtype=0x2b4bc3fa9ea0,
> > > > > comm=0x12cd98c0) at palltoall.c:84
> > > > > #9  0x2b4bc399cc86 in mpi_alltoall_f (sendbuf=0x2aaad10f9d10 "Z\n
> > > > > \271\356\023\254\271?", sendcount=0x7fff33372688, 
> > > > > sendtype=, recvbuf=0x2aaae104d010 "",
> > > > > recvcount=0x7fff3337268c, recvtype=0xb67490, 
> > > > > comm=0x12d9d20, ierr=0x7fff33372690) at palltoall_f.c:76
> > > > > #10 0x004613b8 in m_alltoall_z_ ()
> > > > > #11 0x004ec55f in redis_pw_ ()
> > > > > #12 0x005643d0 in choleski_mp_orthch_ ()
> > > > > #13 0x0043fbba in MAIN__ ()
> > > > > #14 0x0042f15c in main ()
> > > > > 
> > > > > On Tue, 2010-07-27 at 06:14 -0400, Terry Dontje wrote:
> > > > >   
> > > > >   
> > > > >

Re: [OMPI users] openMPI shared with NFS, but says different version

[Cristobal Navarro]

On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa  wrote:

> Hi Cristobal
>
> In case you are not using full path name for mpiexec/mpirun,
> what does "which mpirun" say?
>

--> $which mpirun
  /opt/openmpi-1.4.2

>
> Often times this is a source of confusion, old versions may
> be first on the PATH.
>
> Gus
>

openMPI version problem is now gone, i can confirm that the version is
consistent now :), thanks.

however, i keep getting this kernel crash randomnly when i execute with -np
higher than 5
these are Xeons, with Hyperthreading On, is that a problem??

im trying to locate the kernel error on logs, but after rebooting a crash,
the error is not in the kern.log (neither kern.log.1).
all i remember is that it starts with "Kernel BUG..."
and somepart it mentions a certain CPU X, where that cpu can be any from 0
to 15 (im testing only in main node).  Someone knows where the log of kernel
error could be?

>
> Cristobal Navarro wrote:
>
>>
>> On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa > g...@ldeo.columbia.edu>> wrote:
>>
>>Hi Cristobal
>>
>>Does it run only on the head node alone?
>>(Fuego? Agua? Acatenango?)
>>Try to put only the head node on the hostfile and execute with mpiexec.
>>
>> --> i will try only with the head node, and post results back
>>This may help sort out what is going on.
>>Hopefully it will run on the head node.
>>
>>Also, do you have Infinband connecting the nodes?
>>The error messages refer to the openib btl (i.e. Infiniband),
>>and complains of
>>
>>
>> no we are just using normal network 100MBit/s , since i am just testing
>> yet.
>>
>>
>>"perhaps a missing symbol, or compiled for a different
>>version of Open MPI?".
>>It sounds as a mixup of versions/builds.
>>
>>
>> --> i agree, somewhere there must be the remains of the older version
>>
>>Did you configure/build OpenMPI from source, or did you install
>>it with apt-get?
>>It may be easier/less confusing to install from source.
>>If you did, what configure options did you use?
>>
>>
>> -->i installed from source, ./configure --prefix=/opt/openmpi-1.4.2
>> --with-sge --without-xgid --disable--static
>>
>>Also, as for the OpenMPI runtime environment,
>>it is not enough to set it on
>>the command line, because it will be effective only on the head node.
>>You need to either add them to the PATH and LD_LIBRARY_PATH
>>on your .bashrc/.cshrc files (assuming these files and your home
>>directory are *also* shared with the nodes via NFS),
>>or use the --prefix option of mpiexec to point to the OpenMPI main
>>directory.
>>
>>
>> yes, all nodes have their PATH and LD_LIBRARY_PATH set up properly inside
>> the login scripts ( .bashrc in my case  )
>>
>>Needless to say, you need to check and ensure that the OpenMPI
>>directory (and maybe your home directory, and your work directory)
>>is (are)
>>really mounted on the nodes.
>>
>>
>> --> yes, doublechecked that they are
>>
>>I hope this helps,
>>
>>
>> --> thanks really!
>>
>>Gus Correa
>>
>>Update: i just reinstalled openMPI, with the same parameters, and it
>>seems that the problem has gone, i couldnt test entirely but when i
>>get back to lab ill confirm.
>>
>> best regards! Cristobal
>>
>>
>> 
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

Re: [OMPI users] openMPI shared with NFS, but says different version

[Cristobal Navarro]

to clear things,

i still can do a hello world on all 16 threads, but a few more repetitions
of the example and it kernel crashes :(

fcluster@agua:~$ mpirun --hostfile localhostfile -np 16 testMPI/hola
Process 0 on agua out of 16
Process 2 on agua out of 16
Process 14 on agua out of 16
Process 8 on agua out of 16
Process 1 on agua out of 16
Process 7 on agua out of 16
Process 9 on agua out of 16
Process 3 on agua out of 16
Process 4 on agua out of 16
Process 10 on agua out of 16
Process 15 on agua out of 16
Process 5 on agua out of 16
Process 6 on agua out of 16
Process 11 on agua out of 16
Process 13 on agua out of 16
Process 12 on agua out of 16
fcluster@agua:~$



On Wed, Jul 28, 2010 at 2:47 PM, Cristobal Navarro wrote:

>
>
> On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa wrote:
>
>> Hi Cristobal
>>
>> In case you are not using full path name for mpiexec/mpirun,
>> what does "which mpirun" say?
>>
>
> --> $which mpirun
>   /opt/openmpi-1.4.2
>
>>
>> Often times this is a source of confusion, old versions may
>> be first on the PATH.
>>
>> Gus
>>
>
> openMPI version problem is now gone, i can confirm that the version is
> consistent now :), thanks.
>
> however, i keep getting this kernel crash randomnly when i execute with -np
> higher than 5
> these are Xeons, with Hyperthreading On, is that a problem??
>
> im trying to locate the kernel error on logs, but after rebooting a crash,
> the error is not in the kern.log (neither kern.log.1).
> all i remember is that it starts with "Kernel BUG..."
> and somepart it mentions a certain CPU X, where that cpu can be any from 0
> to 15 (im testing only in main node).  Someone knows where the log of kernel
> error could be?
>
>>
>> Cristobal Navarro wrote:
>>
>>>
>>> On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa >> g...@ldeo.columbia.edu>> wrote:
>>>
>>>Hi Cristobal
>>>
>>>Does it run only on the head node alone?
>>>(Fuego? Agua? Acatenango?)
>>>Try to put only the head node on the hostfile and execute with
>>> mpiexec.
>>>
>>> --> i will try only with the head node, and post results back
>>>This may help sort out what is going on.
>>>Hopefully it will run on the head node.
>>>
>>>Also, do you have Infinband connecting the nodes?
>>>The error messages refer to the openib btl (i.e. Infiniband),
>>>and complains of
>>>
>>>
>>> no we are just using normal network 100MBit/s , since i am just testing
>>> yet.
>>>
>>>
>>>"perhaps a missing symbol, or compiled for a different
>>>version of Open MPI?".
>>>It sounds as a mixup of versions/builds.
>>>
>>>
>>> --> i agree, somewhere there must be the remains of the older version
>>>
>>>Did you configure/build OpenMPI from source, or did you install
>>>it with apt-get?
>>>It may be easier/less confusing to install from source.
>>>If you did, what configure options did you use?
>>>
>>>
>>> -->i installed from source, ./configure --prefix=/opt/openmpi-1.4.2
>>> --with-sge --without-xgid --disable--static
>>>
>>>Also, as for the OpenMPI runtime environment,
>>>it is not enough to set it on
>>>the command line, because it will be effective only on the head node.
>>>You need to either add them to the PATH and LD_LIBRARY_PATH
>>>on your .bashrc/.cshrc files (assuming these files and your home
>>>directory are *also* shared with the nodes via NFS),
>>>or use the --prefix option of mpiexec to point to the OpenMPI main
>>>directory.
>>>
>>>
>>> yes, all nodes have their PATH and LD_LIBRARY_PATH set up properly inside
>>> the login scripts ( .bashrc in my case  )
>>>
>>>Needless to say, you need to check and ensure that the OpenMPI
>>>directory (and maybe your home directory, and your work directory)
>>>is (are)
>>>really mounted on the nodes.
>>>
>>>
>>> --> yes, doublechecked that they are
>>>
>>>I hope this helps,
>>>
>>>
>>> --> thanks really!
>>>
>>>Gus Correa
>>>
>>>Update: i just reinstalled openMPI, with the same parameters, and it
>>>seems that the problem has gone, i couldnt test entirely but when i
>>>get back to lab ill confirm.
>>>
>>> best regards! Cristobal
>>>
>>>
>>> 
>>>
>>> ___
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>

Re: [OMPI users] openMPI shared with NFS, but says different version

[Gus Correa]

Hi Cristobal

Cristobal Navarro wrote:



On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa > wrote:


Hi Cristobal

In case you are not using full path name for mpiexec/mpirun,
what does "which mpirun" say?


--> $which mpirun
  /opt/openmpi-1.4.2


Often times this is a source of confusion, old versions may
be first on the PATH.

Gus


openMPI version problem is now gone, i can confirm that the version is 
consistent now :), thanks.




This is good news.

however, i keep getting this kernel crash randomnly when i execute with 
-np higher than 5

these are Xeons, with Hyperthreading On, is that a problem??



The problem may be with Hyperthreading, maybe not.
Which Xeons?
If I remember right, the old hyperthreading on old Xeons was problematic.

OTOH, about 1-2 months ago I had trouble with OpenMPI on a relatively 
new Xeon Nehalem machine with (the new) Hyperthreading turned on,

and Fedora Core 13.
The machine would hang with the OpenMPI connectivity example.
I reported this to the list, you may find in the archives.
Apparently other people got everything (OpenMPI with HT on Nehalem)
working in more stable distributions (CentOS, RHEL, etc).

That problem was likely to be in the FC13 kernel,
because even turning off HT I still had the machine hanging.
Nothing worked with shared memory turned on,
so I had to switch OpenMPI to use tcp instead,
which is kind of ridiculous in a standalone machine.


im trying to locate the kernel error on logs, but after rebooting a 
crash, the error is not in the kern.log (neither kern.log.1).

all i remember is that it starts with "Kernel BUG..."
and somepart it mentions a certain CPU X, where that cpu can be any from 
0 to 15 (im testing only in main node).  Someone knows where the log of 
kernel error could be?




Have you tried to turn off hyperthreading?
In any case, depending on the application, it may not help much 
performance to have HT on.


A more radical alternative is to try
-mca btl tcp,self
in the mpirun command line.
That is what worked in the case I mentioned above.

My $0.02
Gus Correa



Cristobal Navarro wrote:


On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa
mailto:g...@ldeo.columbia.edu>
>>
wrote:

   Hi Cristobal

   Does it run only on the head node alone?
   (Fuego? Agua? Acatenango?)
   Try to put only the head node on the hostfile and execute
with mpiexec.

--> i will try only with the head node, and post results back
   This may help sort out what is going on.
   Hopefully it will run on the head node.

   Also, do you have Infinband connecting the nodes?
   The error messages refer to the openib btl (i.e. Infiniband),
   and complains of


no we are just using normal network 100MBit/s , since i am just
testing yet.


   "perhaps a missing symbol, or compiled for a different
   version of Open MPI?".
   It sounds as a mixup of versions/builds.


--> i agree, somewhere there must be the remains of the older
version

   Did you configure/build OpenMPI from source, or did you install
   it with apt-get?
   It may be easier/less confusing to install from source.
   If you did, what configure options did you use?


-->i installed from source, ./configure
--prefix=/opt/openmpi-1.4.2 --with-sge --without-xgid
--disable--static

   Also, as for the OpenMPI runtime environment,
   it is not enough to set it on
   the command line, because it will be effective only on the
head node.
   You need to either add them to the PATH and LD_LIBRARY_PATH
   on your .bashrc/.cshrc files (assuming these files and your home
   directory are *also* shared with the nodes via NFS),
   or use the --prefix option of mpiexec to point to the OpenMPI
main
   directory.


yes, all nodes have their PATH and LD_LIBRARY_PATH set up
properly inside the login scripts ( .bashrc in my case  )

   Needless to say, you need to check and ensure that the OpenMPI
   directory (and maybe your home directory, and your work
directory)
   is (are)
   really mounted on the nodes.


--> yes, doublechecked that they are

   I hope this helps,


--> thanks really!

   Gus Correa

   Update: i just reinstalled openMPI, with the same parameters,
and it
   seems that the problem has gone, i couldnt test entirely but
when i
   get back to lab ill confirm.

best regards! Cristobal




___
users mailing list

Re: [OMPI users] openMPI shared with NFS, but says different version

[Cristobal Navarro]

On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa  wrote:

> Hi Cristobal
>
> Cristobal Navarro wrote:
>
>>
>>
>> On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa > g...@ldeo.columbia.edu>> wrote:
>>
>>Hi Cristobal
>>
>>In case you are not using full path name for mpiexec/mpirun,
>>what does "which mpirun" say?
>>
>>
>> --> $which mpirun
>>  /opt/openmpi-1.4.2
>>
>>
>>Often times this is a source of confusion, old versions may
>>be first on the PATH.
>>
>>Gus
>>
>>
>> openMPI version problem is now gone, i can confirm that the version is
>> consistent now :), thanks.
>>
>>
> This is good news.
>
>
>  however, i keep getting this kernel crash randomnly when i execute with
>> -np higher than 5
>> these are Xeons, with Hyperthreading On, is that a problem??
>>
>>
> The problem may be with Hyperthreading, maybe not.
> Which Xeons?
>

--> they are not so old, not so new either
fcluster@agua:~$ cat /proc/cpuinfo | more
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU   E5520  @ 2.27GHz
stepping : 5
cpu MHz : 1596.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts acpi mmx fxsr sse sse2 ss h
t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_
cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida
tpr_shadow vnmi flexpriority ept vpid
bogomips : 4522.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
...same for cpu1, 2, 3, ..., 15.

information on how the cpu is distributed
fcluster@agua:~$ lstopo
System(7992MB)
  Socket#0 + L3(8192KB)
L2(256KB) + L1(32KB) + Core#0
  P#0
  P#8
L2(256KB) + L1(32KB) + Core#1
  P#2
  P#10
L2(256KB) + L1(32KB) + Core#2
  P#4
  P#12
L2(256KB) + L1(32KB) + Core#3
  P#6
  P#14
  Socket#1 + L3(8192KB)
L2(256KB) + L1(32KB) + Core#0
  P#1
  P#9
L2(256KB) + L1(32KB) + Core#1
  P#3
  P#11
L2(256KB) + L1(32KB) + Core#2
  P#5
  P#13
L2(256KB) + L1(32KB) + Core#3
  P#7
  P#15





> If I remember right, the old hyperthreading on old Xeons was problematic.
>
> OTOH, about 1-2 months ago I had trouble with OpenMPI on a relatively new
> Xeon Nehalem machine with (the new) Hyperthreading turned on,
> and Fedora Core 13.
> The machine would hang with the OpenMPI connectivity example.
> I reported this to the list, you may find in the archives.
>

--i foudn the archives recently about an hour ago, was not sure if it was
the same problem but i removed HT for testing with setting the online flag
to 0 on the extra cpus showed with lstopo, unfortenately i also crashes, so
HT may not be the problem.

> Apparently other people got everything (OpenMPI with HT on Nehalem)
> working in more stable distributions (CentOS, RHEL, etc).
>
> That problem was likely to be in the FC13 kernel,
> because even turning off HT I still had the machine hanging.
> Nothing worked with shared memory turned on,
> so I had to switch OpenMPI to use tcp instead,
> which is kind of ridiculous in a standalone machine.


--> very interesting, sm can be the problem


>
>
>
>  im trying to locate the kernel error on logs, but after rebooting a crash,
>> the error is not in the kern.log (neither kern.log.1).
>> all i remember is that it starts with "Kernel BUG..."
>> and somepart it mentions a certain CPU X, where that cpu can be any from 0
>> to 15 (im testing only in main node).  Someone knows where the log of kernel
>> error could be?
>>
>>
> Have you tried to turn off hyperthreading?
>

--> yes, tried, same crashes.


> In any case, depending on the application, it may not help much performance
> to have HT on.
>
> A more radical alternative is to try
> -mca btl tcp,self
> in the mpirun command line.
> That is what worked in the case I mentioned above.
>

wow!, this worked really :),  you pointed out the problem, it was shared
memory.
i have 4 nodes, so anyways there will be node comunication, do you think i
can rely on working with -mca btl tcp,self?? i dont mind small lag.

i just have one more question, is this a problem of the ubuntu server
kernel?? from the Nehalem Cpus?? from openMPI (i dont think) ??

and on what depends that in the future, sm could be possible on the same
configuration i have?? kernel update?.

Thanks very much Gus, really!
Cristobal




>
> My $0.02
> Gus Correa
>
>
>>Cristobal Navarro wrote:
>>
>>
>>On Tue, Jul 27, 2010 at 7:29 PM, Gus Correa
>>mailto:g...@ldeo.columbia.edu>
>>>>
>>
>>wrote:
>>
>>   Hi Cristobal
>>
>>   Does it run only on the head node alone?
>>   

Re: [OMPI users] MPI_Allreduce on local machine

[Martin Siegert]

On Wed, Jul 28, 2010 at 11:19:43AM -0400, Gus Correa wrote:
> Hugo Gagnon wrote:
>> Hi Gus,
>> Ompi_info --all lists its info regarding fortran right after C. In my
>> case:
>>   Fort real size: 4
>>  Fort real4 size: 4
>>  Fort real8 size: 8
>> Fort real16 size: 16
>>   Fort dbl prec size: 4
>> Does it make any sense to you?
>
> Hi Hugo
>
> No, dbl prec size 4 sounds weird, should be 8, I suppose,
> same as real8, right?
>
> It doesn't make sense, but that's what I have (now that you told me
> that "dbl" , not "double", is the string to search for):
>
> $  Fort dbl prec size: 4
>   Fort dbl cplx size: 4
>  Fort dbl prec align: 4
>  Fort dbl cplx align: 4
>
> Is this a bug in OpenMPI perhaps?
>
> I didn't come across to this problem, most likely because
> the codes here don't use "double precision" but real*8 or similar.
>
> Also make sure you are picking the right ompi_info, mpif90/f77, mpiexec.
> Often times old versions and tangled PATH make things very confusing.

This is indeed worrisome as I confirm the findings on our clusters both
with ompi 1.3.3 and 1.4.1:

ompi_info --all | grep -i fort
...
  Fort real size: 4
 Fort real4 size: 4
 Fort real8 size: 8
Fort real16 size: -1
  Fort dbl prec size: 4
  Fort cplx size: 4
  Fort dbl cplx size: 4
 Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
  Fort integer align: 4
 Fort integer1 align: 1
 Fort integer2 align: 2
 Fort integer4 align: 4
 Fort integer8 align: 8
Fort integer16 align: -1
 Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
   Fort real16 align: -1
 Fort dbl prec align: 4
 Fort cplx align: 4  
 Fort dbl cplx align: 4  
Fort cplx8 align: 4  
   Fort cplx16 align: 8  
...

And this is the configure output:
checking if Fortran 77 compiler supports REAL*8... yes
checking size of Fortran 77 REAL*8... 8
checking for C type corresponding to REAL*8... double
checking alignment of Fortran REAL*8... 1
...
checking if Fortran 77 compiler supports DOUBLE PRECISION... yes
checking size of Fortran 77 DOUBLE PRECISION... 8
checking for C type corresponding to DOUBLE PRECISION... double
checking alignment of Fortran DOUBLE PRECISION... 1

But the following code actually appears to give the correct results:

program types
use mpi
implicit none
integer :: mpierr, size

   call MPI_Init(mpierr)
   call MPI_Type_size(MPI_DOUBLE_PRECISION, size, mpierr)
   print*, 'double precision size: ', size
   call MPI_Finalize(mpierr)
end

mpif90 -g types.f90
mpiexec -n 1 ./a.out
 double precision size:8

Thus is this a bug in ompi_info only?

Cheers,
Martin

-- 
Martin Siegert
Head, Research Computing
WestGrid/ComputeCanada Site Lead
IT Servicesphone: 778 782-4691
Simon Fraser Universityfax:   778 782-4242
Burnaby, British Columbia  email: sieg...@sfu.ca
Canada  V5A 1S6

Re: [OMPI users] MPI_Allreduce on local machine

[Martin Siegert]

On Wed, Jul 28, 2010 at 01:05:52PM -0700, Martin Siegert wrote:
> On Wed, Jul 28, 2010 at 11:19:43AM -0400, Gus Correa wrote:
> > Hugo Gagnon wrote:
> >> Hi Gus,
> >> Ompi_info --all lists its info regarding fortran right after C. In my
> >> case:
> >>   Fort real size: 4
> >>  Fort real4 size: 4
> >>  Fort real8 size: 8
> >> Fort real16 size: 16
> >>   Fort dbl prec size: 4
> >> Does it make any sense to you?
> >
> > Hi Hugo
> >
> > No, dbl prec size 4 sounds weird, should be 8, I suppose,
> > same as real8, right?
> >
> > It doesn't make sense, but that's what I have (now that you told me
> > that "dbl" , not "double", is the string to search for):
> >
> > $  Fort dbl prec size: 4
> >   Fort dbl cplx size: 4
> >  Fort dbl prec align: 4
> >  Fort dbl cplx align: 4
> >
> > Is this a bug in OpenMPI perhaps?
> >
> > I didn't come across to this problem, most likely because
> > the codes here don't use "double precision" but real*8 or similar.
> >
> > Also make sure you are picking the right ompi_info, mpif90/f77, mpiexec.
> > Often times old versions and tangled PATH make things very confusing.
> 
> This is indeed worrisome as I confirm the findings on our clusters both
> with ompi 1.3.3 and 1.4.1:
> 
> ompi_info --all | grep -i fort
> ...
>   Fort real size: 4
>  Fort real4 size: 4
>  Fort real8 size: 8
> Fort real16 size: -1
>   Fort dbl prec size: 4
>   Fort cplx size: 4
>   Fort dbl cplx size: 4
>  Fort cplx8 size: 8
> Fort cplx16 size: 16
> Fort cplx32 size: -1
>   Fort integer align: 4
>  Fort integer1 align: 1
>  Fort integer2 align: 2
>  Fort integer4 align: 4
>  Fort integer8 align: 8
> Fort integer16 align: -1
>  Fort real align: 4
> Fort real4 align: 4
> Fort real8 align: 8
>Fort real16 align: -1
>  Fort dbl prec align: 4
>  Fort cplx align: 4  
>  Fort dbl cplx align: 4  
> Fort cplx8 align: 4  
>Fort cplx16 align: 8  
> ...
> 
> And this is the configure output:
> checking if Fortran 77 compiler supports REAL*8... yes
> checking size of Fortran 77 REAL*8... 8
> checking for C type corresponding to REAL*8... double
> checking alignment of Fortran REAL*8... 1
> ...
> checking if Fortran 77 compiler supports DOUBLE PRECISION... yes
> checking size of Fortran 77 DOUBLE PRECISION... 8
> checking for C type corresponding to DOUBLE PRECISION... double
> checking alignment of Fortran DOUBLE PRECISION... 1
> 
> But the following code actually appears to give the correct results:
> 
> program types
> use mpi
> implicit none
> integer :: mpierr, size
> 
>call MPI_Init(mpierr)
>call MPI_Type_size(MPI_DOUBLE_PRECISION, size, mpierr)
>print*, 'double precision size: ', size
>call MPI_Finalize(mpierr)
> end
> 
> mpif90 -g types.f90
> mpiexec -n 1 ./a.out
>  double precision size:8
> 
> Thus is this a bug in ompi_info only?

answering my own question:
This does not look right:

ompi/tools/ompi_info/param.cc:

  out("Fort dbl prec size",
  "compiler:fortran:sizeof:double_precision",
  OMPI_SIZEOF_FORTRAN_REAL);

that should be OMPI_SIZEOF_FORTRAN_DOUBLE_PRECISION.

- Martin

Re: [OMPI users] MPI_Allreduce on local machine

[Gus Correa]

Hi All

Martin Siegert wrote:

On Wed, Jul 28, 2010 at 01:05:52PM -0700, Martin Siegert wrote:

On Wed, Jul 28, 2010 at 11:19:43AM -0400, Gus Correa wrote:

Hugo Gagnon wrote:

Hi Gus,
Ompi_info --all lists its info regarding fortran right after C. In my
case:
  Fort real size: 4
 Fort real4 size: 4
 Fort real8 size: 8
Fort real16 size: 16
  Fort dbl prec size: 4
Does it make any sense to you?

Hi Hugo

No, dbl prec size 4 sounds weird, should be 8, I suppose,
same as real8, right?

It doesn't make sense, but that's what I have (now that you told me
that "dbl" , not "double", is the string to search for):

$  Fort dbl prec size: 4
  Fort dbl cplx size: 4
 Fort dbl prec align: 4
 Fort dbl cplx align: 4

Is this a bug in OpenMPI perhaps?

I didn't come across to this problem, most likely because
the codes here don't use "double precision" but real*8 or similar.

Also make sure you are picking the right ompi_info, mpif90/f77, mpiexec.
Often times old versions and tangled PATH make things very confusing.

This is indeed worrisome as I confirm the findings on our clusters both
with ompi 1.3.3 and 1.4.1:

ompi_info --all | grep -i fort
...
  Fort real size: 4
 Fort real4 size: 4
 Fort real8 size: 8
Fort real16 size: -1
  Fort dbl prec size: 4
  Fort cplx size: 4
  Fort dbl cplx size: 4
 Fort cplx8 size: 8
Fort cplx16 size: 16
Fort cplx32 size: -1
  Fort integer align: 4
 Fort integer1 align: 1
 Fort integer2 align: 2
 Fort integer4 align: 4
 Fort integer8 align: 8
Fort integer16 align: -1
 Fort real align: 4
Fort real4 align: 4
Fort real8 align: 8
   Fort real16 align: -1
 Fort dbl prec align: 4
 Fort cplx align: 4  
 Fort dbl cplx align: 4  
Fort cplx8 align: 4  
   Fort cplx16 align: 8  
...


And this is the configure output:
checking if Fortran 77 compiler supports REAL*8... yes
checking size of Fortran 77 REAL*8... 8
checking for C type corresponding to REAL*8... double
checking alignment of Fortran REAL*8... 1
...
checking if Fortran 77 compiler supports DOUBLE PRECISION... yes
checking size of Fortran 77 DOUBLE PRECISION... 8
checking for C type corresponding to DOUBLE PRECISION... double
checking alignment of Fortran DOUBLE PRECISION... 1

But the following code actually appears to give the correct results:

program types
use mpi
implicit none
integer :: mpierr, size

   call MPI_Init(mpierr)
   call MPI_Type_size(MPI_DOUBLE_PRECISION, size, mpierr)
   print*, 'double precision size: ', size
   call MPI_Finalize(mpierr)
end

mpif90 -g types.f90
mpiexec -n 1 ./a.out
 double precision size:8

Thus is this a bug in ompi_info only?


answering my own question:
This does not look right:

ompi/tools/ompi_info/param.cc:

  out("Fort dbl prec size",
  "compiler:fortran:sizeof:double_precision",
  OMPI_SIZEOF_FORTRAN_REAL);

that should be OMPI_SIZEOF_FORTRAN_DOUBLE_PRECISION.

- Martin


Hopefully Martin may got it and the issue is restricted to ompi_info.
Thanks, Martin, for writing and running the little diagnostic code,
and for checking the ompi_info guts!

Still, the alignment under Intel may or may not be right.
And this may or may not explain the errors that Hugo has got.

FYI, the ompi_info from my OpenMPI 1.3.2 and 1.2.8
report exactly the same as OpenMPI 1.4.2, namely
Fort dbl prec size: 4  and
Fort dbl prec align: 4,
except that *if the Intel Fortran compiler (ifort) was used*
I get 1 byte alignment:
Fort dbl prec align: 1

So, this issue has been around for a while,
and involves both the size and the alignment (in Intel)
of double precision.

We have a number of pieces of code here where grep shows 
MPI_DOUBLE_PRECISION.

Not sure how much of it has actually been active, as there are always
lots of cpp directives to select active code.

In particular I found this interesting snippet:

if (MPI_DOUBLE_PRECISION==20 .and. MPI_REAL8==18) then
   ! LAM MPI defined MPI_REAL8 differently from MPI_DOUBLE_PRECISION
   ! and LAM MPI's allreduce does not accept on MPI_REAL8
   MPIreal_t= MPI_DOUBLE_PRECISION
else
   MPIreal_t= MPI_REAL8
endif

where eventually MPIreal_t is what is used as
the MPI type in some MPI calls, particularly in MPI_Allreduce,
which is the one that triggered all this discussion
(see this thread Subject line) when Hugo first
asked his original question.

Hopefully the if branch on the code snippet above worked alright,
because here in our OpenMPIs 1.4.2, 1.3.2, and 1.2.8,
MPI_DOUBLE_PRECISION value is 17,
which should have safely produced
MPIreal_t= MPI_REAL8

I have a lot more of code to check, but maybe not.
If the issue is really restricted to ompi_info that would be a
big relief.

Many th

Re: [OMPI users] MPI_Allreduce on local machine

[Hugo Gagnon]

I also get 8 from "call MPI_Type_size(MPI_DOUBLE_PRECISION, size,
mpierr)", but really I don't think this is the issue anymore. I mean I
checked on my school cluster where OpenMPI has also been compiled with
the intel64 compilers and "Fort dbl prec size:" also returns 4 but
unlike on my Mac the code runs fine there. I am just saying that we
should stop worrying about ompi_info output and wait until Jeff Squyres
analyses my build output files that I sent to the list earlier. I might
be wrong too as I have no idea of what's going on.
-- 
  Hugo Gagnon


On Wed, 28 Jul 2010 17:07 -0400, "Gus Correa" 
wrote:
> Hi All
> 
> Martin Siegert wrote:
> > On Wed, Jul 28, 2010 at 01:05:52PM -0700, Martin Siegert wrote:
> >> On Wed, Jul 28, 2010 at 11:19:43AM -0400, Gus Correa wrote:
> >>> Hugo Gagnon wrote:
>  Hi Gus,
>  Ompi_info --all lists its info regarding fortran right after C. In my
>  case:
>    Fort real size: 4
>   Fort real4 size: 4
>   Fort real8 size: 8
>  Fort real16 size: 16
>    Fort dbl prec size: 4
>  Does it make any sense to you?
> >>> Hi Hugo
> >>>
> >>> No, dbl prec size 4 sounds weird, should be 8, I suppose,
> >>> same as real8, right?
> >>>
> >>> It doesn't make sense, but that's what I have (now that you told me
> >>> that "dbl" , not "double", is the string to search for):
> >>>
> >>> $  Fort dbl prec size: 4
> >>>   Fort dbl cplx size: 4
> >>>  Fort dbl prec align: 4
> >>>  Fort dbl cplx align: 4
> >>>
> >>> Is this a bug in OpenMPI perhaps?
> >>>
> >>> I didn't come across to this problem, most likely because
> >>> the codes here don't use "double precision" but real*8 or similar.
> >>>
> >>> Also make sure you are picking the right ompi_info, mpif90/f77, mpiexec.
> >>> Often times old versions and tangled PATH make things very confusing.
> >> This is indeed worrisome as I confirm the findings on our clusters both
> >> with ompi 1.3.3 and 1.4.1:
> >>
> >> ompi_info --all | grep -i fort
> >> ...
> >>   Fort real size: 4
> >>  Fort real4 size: 4
> >>  Fort real8 size: 8
> >> Fort real16 size: -1
> >>   Fort dbl prec size: 4
> >>   Fort cplx size: 4
> >>   Fort dbl cplx size: 4
> >>  Fort cplx8 size: 8
> >> Fort cplx16 size: 16
> >> Fort cplx32 size: -1
> >>   Fort integer align: 4
> >>  Fort integer1 align: 1
> >>  Fort integer2 align: 2
> >>  Fort integer4 align: 4
> >>  Fort integer8 align: 8
> >> Fort integer16 align: -1
> >>  Fort real align: 4
> >> Fort real4 align: 4
> >> Fort real8 align: 8
> >>Fort real16 align: -1
> >>  Fort dbl prec align: 4
> >>  Fort cplx align: 4  
> >>  Fort dbl cplx align: 4  
> >> Fort cplx8 align: 4  
> >>Fort cplx16 align: 8  
> >> ...
> >>
> >> And this is the configure output:
> >> checking if Fortran 77 compiler supports REAL*8... yes
> >> checking size of Fortran 77 REAL*8... 8
> >> checking for C type corresponding to REAL*8... double
> >> checking alignment of Fortran REAL*8... 1
> >> ...
> >> checking if Fortran 77 compiler supports DOUBLE PRECISION... yes
> >> checking size of Fortran 77 DOUBLE PRECISION... 8
> >> checking for C type corresponding to DOUBLE PRECISION... double
> >> checking alignment of Fortran DOUBLE PRECISION... 1
> >>
> >> But the following code actually appears to give the correct results:
> >>
> >> program types
> >> use mpi
> >> implicit none
> >> integer :: mpierr, size
> >>
> >>call MPI_Init(mpierr)
> >>call MPI_Type_size(MPI_DOUBLE_PRECISION, size, mpierr)
> >>print*, 'double precision size: ', size
> >>call MPI_Finalize(mpierr)
> >> end
> >>
> >> mpif90 -g types.f90
> >> mpiexec -n 1 ./a.out
> >>  double precision size:8
> >>
> >> Thus is this a bug in ompi_info only?
> > 
> > answering my own question:
> > This does not look right:
> > 
> > ompi/tools/ompi_info/param.cc:
> > 
> >   out("Fort dbl prec size",
> >   "compiler:fortran:sizeof:double_precision",
> >   OMPI_SIZEOF_FORTRAN_REAL);
> > 
> > that should be OMPI_SIZEOF_FORTRAN_DOUBLE_PRECISION.
> > 
> > - Martin
> 
> Hopefully Martin may got it and the issue is restricted to ompi_info.
> Thanks, Martin, for writing and running the little diagnostic code,
> and for checking the ompi_info guts!
> 
> Still, the alignment under Intel may or may not be right.
> And this may or may not explain the errors that Hugo has got.
> 
> FYI, the ompi_info from my OpenMPI 1.3.2 and 1.2.8
> report exactly the same as OpenMPI 1.4.2, namely
> Fort dbl prec size: 4  and
> Fort dbl prec align: 4,
> except that *if the Intel Fortran compiler (ifort) was used*
> I get 1 byte alignment:
> Fort dbl prec align: 1
> 
> So, this issue has been around for a while,
> and involves b

Re: [OMPI users] openMPI shared with NFS, but says different version

[Gus Correa]

Hi Cristobal

Please, read my answer (way down the message) below.

Cristobal Navarro wrote:



On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa > wrote:


Hi Cristobal

Cristobal Navarro wrote:



On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa
mailto:g...@ldeo.columbia.edu>
>>
wrote:

   Hi Cristobal

   In case you are not using full path name for mpiexec/mpirun,
   what does "which mpirun" say?


--> $which mpirun
 /opt/openmpi-1.4.2


   Often times this is a source of confusion, old versions may
   be first on the PATH.

   Gus


openMPI version problem is now gone, i can confirm that the
version is consistent now :), thanks.


This is good news.


however, i keep getting this kernel crash randomnly when i
execute with -np higher than 5
these are Xeons, with Hyperthreading On, is that a problem??


The problem may be with Hyperthreading, maybe not.
Which Xeons?


--> they are not so old, not so new either
fcluster@agua:~$ cat /proc/cpuinfo | more
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU   E5520  @ 2.27GHz
stepping : 5
cpu MHz : 1596.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss h
t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts 
rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_
cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm 
ida tpr_shadow vnmi flexpriority ept vpid

bogomips : 4522.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
...same for cpu1, 2, 3, ..., 15.



AHA! Nehalems!

Here they are E5540, just a different clock speed, I suppose.


information on how the cpu is distributed
fcluster@agua:~$ lstopo
System(7992MB)
  Socket#0 + L3(8192KB)
L2(256KB) + L1(32KB) + Core#0
  P#0
  P#8
L2(256KB) + L1(32KB) + Core#1
  P#2
  P#10
L2(256KB) + L1(32KB) + Core#2
  P#4
  P#12
L2(256KB) + L1(32KB) + Core#3
  P#6
  P#14
  Socket#1 + L3(8192KB)
L2(256KB) + L1(32KB) + Core#0
  P#1
  P#9
L2(256KB) + L1(32KB) + Core#1
  P#3
  P#11
L2(256KB) + L1(32KB) + Core#2
  P#5
  P#13
L2(256KB) + L1(32KB) + Core#3
  P#7
  P#15



 


If I remember right, the old hyperthreading on old Xeons was
problematic.

OTOH, about 1-2 months ago I had trouble with OpenMPI on a
relatively new Xeon Nehalem machine with (the new) Hyperthreading
turned on,
and Fedora Core 13.
The machine would hang with the OpenMPI connectivity example.
I reported this to the list, you may find in the archives.


--i foudn the archives recently about an hour ago, was not sure if it 
was the same problem but i removed HT for testing with setting the 
online flag to 0 on the extra cpus showed with lstopo, unfortenately i 
also crashes, so HT may not be the problem. 



It didn't fix the problem in our Nehalem machine here either,
although it was FC13, and I don't know what OS and kernel you're using.


Apparently other people got everything (OpenMPI with HT on Nehalem)
working in more stable distributions (CentOS, RHEL, etc).

That problem was likely to be in the FC13 kernel,
because even turning off HT I still had the machine hanging.
Nothing worked with shared memory turned on,
so I had to switch OpenMPI to use tcp instead,
which is kind of ridiculous in a standalone machine.


--> very interesting, sm can be the problem
 





im trying to locate the kernel error on logs, but after
rebooting a crash, the error is not in the kern.log (neither
kern.log.1).
all i remember is that it starts with "Kernel BUG..."
and somepart it mentions a certain CPU X, where that cpu can be
any from 0 to 15 (im testing only in main node).  Someone knows
where the log of kernel error could be?


Have you tried to turn off hyperthreading?


--> yes, tried, same crashes.
 


In any case, depending on the application, it may not help much
performance to have HT on.

A more radical alternative is to try
-mca btl tcp,self
in the mpirun command line.
That is what worked in the case I mentioned above.


wow!, this worked really :),  you pointed out the problem, it was shared 
memory.


Great news!
That's exactly the problem we had here.
Glad that the same solution worked for you.

Over a year ago another fellow reported the same problem on Nehalem,
on the very early days of Nehalem.
The threa

Re: [OMPI users] openMPI shared with NFS, but says different version

[Cristobal Navarro]

Gus
my kernel for all nodes is this one:
Linux 2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64
GNU/Linux

at least for the moment i will use this configuration, at least for
deveplopment/testing  of the parallel programs.
lag is minimum :)

whenever i get another kernel update, i will test again to check if sm
works, would be good to know that suddenly another distribution supports
nehalem sm.

best regards and thanks again
Cristobal
ps: guess what are the names of the other 2 nodes lol



On Wed, Jul 28, 2010 at 5:50 PM, Gus Correa  wrote:

> Hi Cristobal
>
> Please, read my answer (way down the message) below.
>
> Cristobal Navarro wrote:
>
>>
>>
>> On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa > g...@ldeo.columbia.edu>> wrote:
>>
>>Hi Cristobal
>>
>>Cristobal Navarro wrote:
>>
>>
>>
>>On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa
>>mailto:g...@ldeo.columbia.edu>
>>>>
>>wrote:
>>
>>   Hi Cristobal
>>
>>   In case you are not using full path name for mpiexec/mpirun,
>>   what does "which mpirun" say?
>>
>>
>>--> $which mpirun
>> /opt/openmpi-1.4.2
>>
>>
>>   Often times this is a source of confusion, old versions may
>>   be first on the PATH.
>>
>>   Gus
>>
>>
>>openMPI version problem is now gone, i can confirm that the
>>version is consistent now :), thanks.
>>
>>
>>This is good news.
>>
>>
>>however, i keep getting this kernel crash randomnly when i
>>execute with -np higher than 5
>>these are Xeons, with Hyperthreading On, is that a problem??
>>
>>
>>The problem may be with Hyperthreading, maybe not.
>>Which Xeons?
>>
>>
>> --> they are not so old, not so new either
>> fcluster@agua:~$ cat /proc/cpuinfo | more
>> processor : 0
>> vendor_id : GenuineIntel
>> cpu family : 6
>> model : 26
>> model name : Intel(R) Xeon(R) CPU   E5520  @ 2.27GHz
>> stepping : 5
>> cpu MHz : 1596.000
>> cache size : 8192 KB
>> physical id : 0
>> siblings : 8
>> core id : 0
>> cpu cores : 4
>> apicid : 0
>> initial apicid : 0
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 11
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
>> pse36 clflush dts acpi mmx fxsr sse sse2 ss h
>> t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>> xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_
>> cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida
>> tpr_shadow vnmi flexpriority ept vpid
>> bogomips : 4522.21
>> clflush size : 64
>> cache_alignment : 64
>> address sizes : 40 bits physical, 48 bits virtual
>> power management:
>> ...same for cpu1, 2, 3, ..., 15.
>>
>>
> AHA! Nehalems!
>
> Here they are E5540, just a different clock speed, I suppose.
>
>
>  information on how the cpu is distributed
>> fcluster@agua:~$ lstopo
>> System(7992MB)
>>  Socket#0 + L3(8192KB)
>>L2(256KB) + L1(32KB) + Core#0
>>  P#0
>>  P#8
>>L2(256KB) + L1(32KB) + Core#1
>>  P#2
>>  P#10
>>L2(256KB) + L1(32KB) + Core#2
>>  P#4
>>  P#12
>>L2(256KB) + L1(32KB) + Core#3
>>  P#6
>>  P#14
>>  Socket#1 + L3(8192KB)
>>L2(256KB) + L1(32KB) + Core#0
>>  P#1
>>  P#9
>>L2(256KB) + L1(32KB) + Core#1
>>  P#3
>>  P#11
>>L2(256KB) + L1(32KB) + Core#2
>>  P#5
>>  P#13
>>L2(256KB) + L1(32KB) + Core#3
>>  P#7
>>  P#15
>>
>>
>>
>>
>>If I remember right, the old hyperthreading on old Xeons was
>>problematic.
>>
>>OTOH, about 1-2 months ago I had trouble with OpenMPI on a
>>relatively new Xeon Nehalem machine with (the new) Hyperthreading
>>turned on,
>>and Fedora Core 13.
>>The machine would hang with the OpenMPI connectivity example.
>>I reported this to the list, you may find in the archives.
>>
>>
>> --i foudn the archives recently about an hour ago, was not sure if it was
>> the same problem but i removed HT for testing with setting the online flag
>> to 0 on the extra cpus showed with lstopo, unfortenately i also crashes, so
>> HT may not be the problem.
>>
>
> It didn't fix the problem in our Nehalem machine here either,
> although it was FC13, and I don't know what OS and kernel you're using.
>
>
> Apparently other people got everything (OpenMPI with HT on Nehalem)
>>working in more stable distributions (CentOS, RHEL, etc).
>>
>>That problem was likely to be in the FC13 kernel,
>>because even turning off HT I still had the machine hanging.
>>Nothing worked with shared memory turned on,
>>so I had to switch OpenMPI to use tcp instead,
>>which is kind of ridiculous in a standalone machine.
>>
>>
>> --> very interesting, sm can be the problem
>>
>>
>>
>>
>>im trying to locate the kernel error on logs, but after
>>rebooting a crash, the error is not in the kern.log (neither
>>   

[OMPI users] Alignment of Fortran variables with ifort

[Martin Siegert]

Hi,

I am creating a new thread (was: MPI_Allreduce on local machine).

On Wed, Jul 28, 2010 at 05:07:29PM -0400, Gus Correa wrote:
> Still, the alignment under Intel may or may not be right.
> And this may or may not explain the errors that Hugo has got.
>
> FYI, the ompi_info from my OpenMPI 1.3.2 and 1.2.8
> report exactly the same as OpenMPI 1.4.2, namely
> Fort dbl prec size: 4  and
> Fort dbl prec align: 4,
> except that *if the Intel Fortran compiler (ifort) was used*
> I get 1 byte alignment:
> Fort dbl prec align: 1
>
> So, this issue has been around for a while,
> and involves both the size and the alignment (in Intel)
> of double precision.

There is something strange going on when configuring with ifort.
This is the output from configure:

# grep 'alignment of Fortran' configure-intel.log
checking alignment of Fortran LOGICAL... 1
checking alignment of Fortran LOGICAL*1... 1
checking alignment of Fortran LOGICAL*2... 1
checking alignment of Fortran LOGICAL*4... 1
checking alignment of Fortran LOGICAL*8... 1
checking alignment of Fortran INTEGER... 1
checking alignment of Fortran INTEGER*1... 1
checking alignment of Fortran INTEGER*2... 1
checking alignment of Fortran INTEGER*4... 1
checking alignment of Fortran INTEGER*8... 1
checking alignment of Fortran REAL... 1
checking alignment of Fortran REAL*4... 1
checking alignment of Fortran REAL*8... 1
checking alignment of Fortran REAL*16... 1
checking alignment of Fortran DOUBLE PRECISION... 1
checking alignment of Fortran COMPLEX... 1
checking alignment of Fortran COMPLEX*8... 1
checking alignment of Fortran COMPLEX*16... 1
checking alignment of Fortran COMPLEX*32... 1

All alignments are 1. And this gets entered into opal/include/opal_config.h:

# grep OMPI_ALIGNMENT_FORTRAN opal/include/opal_config.h
#define OMPI_ALIGNMENT_FORTRAN_COMPLEX 1
#define OMPI_ALIGNMENT_FORTRAN_COMPLEX16 1
#define OMPI_ALIGNMENT_FORTRAN_COMPLEX32 1
#define OMPI_ALIGNMENT_FORTRAN_COMPLEX8 1
#define OMPI_ALIGNMENT_FORTRAN_DOUBLE_PRECISION 1
#define OMPI_ALIGNMENT_FORTRAN_INTEGER 1
#define OMPI_ALIGNMENT_FORTRAN_INTEGER1 1
#define OMPI_ALIGNMENT_FORTRAN_INTEGER16 4
#define OMPI_ALIGNMENT_FORTRAN_INTEGER2 1
#define OMPI_ALIGNMENT_FORTRAN_INTEGER4 1
#define OMPI_ALIGNMENT_FORTRAN_INTEGER8 1
#define OMPI_ALIGNMENT_FORTRAN_LOGICAL 1
#define OMPI_ALIGNMENT_FORTRAN_LOGICAL1 1
#define OMPI_ALIGNMENT_FORTRAN_LOGICAL2 1
#define OMPI_ALIGNMENT_FORTRAN_LOGICAL4 1
#define OMPI_ALIGNMENT_FORTRAN_LOGICAL8 1
#define OMPI_ALIGNMENT_FORTRAN_REAL 1
#define OMPI_ALIGNMENT_FORTRAN_REAL16 1
#define OMPI_ALIGNMENT_FORTRAN_REAL2 4
#define OMPI_ALIGNMENT_FORTRAN_REAL4 1
#define OMPI_ALIGNMENT_FORTRAN_REAL8 1

Whereas with gfortran everything appears correctly:

# grep 'alignment of Fortran' configure-gfortran.log
checking alignment of Fortran LOGICAL... 4
checking alignment of Fortran LOGICAL*1... 1
checking alignment of Fortran LOGICAL*2... 2
checking alignment of Fortran LOGICAL*4... 4
checking alignment of Fortran LOGICAL*8... 8
checking alignment of Fortran INTEGER... 4
checking alignment of Fortran INTEGER*1... 1
checking alignment of Fortran INTEGER*2... 2
checking alignment of Fortran INTEGER*4... 4
checking alignment of Fortran INTEGER*8... 8
checking alignment of Fortran REAL... 4
checking alignment of Fortran REAL*4... 4
checking alignment of Fortran REAL*8... 8
checking alignment of Fortran DOUBLE PRECISION... 8
checking alignment of Fortran COMPLEX... 4
checking alignment of Fortran COMPLEX*8... 4
checking alignment of Fortran COMPLEX*16... 8

I may be totally lucky here: I never really use ifort to compile
openmpi - I only use it to compile the mpi.mod module, but use it
with the gfortran "compiled" openmpi
(see:
http://www.open-mpi.org/community/lists/users/2009/07/10017.php
)
As far as I can tell the Fortran compiler is only used by configure
and to compile the Fortran module. It is not used for anything else
when compiling openmpi.

Thus this procedure may have saved me - or not:
a grep through the openmpi code
# grep -r -l --exclude=*.log OMPI_ALIGNMENT_FORTRAN .
./ompi/datatype/dt_module.c
./ompi/tools/ompi_info/param.cc
./opal/include/opal_config.h
shows that there are only three files that reference such variables.
The only one that is worrisome is dt_module.c.
Maybe these macros do not play a role?

- Martin

Re: [OMPI users] openMPI shared with NFS, but says different version

[Gus Correa]

Cristobal Navarro wrote:

Gus
my kernel for all nodes is this one:
Linux 2.6.32-22-server #36-Ubuntu SMP Thu Jun 3 20:38:33 UTC 2010 x86_64 
GNU/Linux




Kernel is not my league.

However, it would be great if somebody clarified
for good these issues with Nehalem/Westmere, HT,
shared memory and what the kernel is doing,
or how to make the kernel do the right thing.
Maybe Intel could tell.

at least for the moment i will use this configuration, at least for 
deveplopment/testing  of the parallel programs.

lag is minimum :)

whenever i get another kernel update, i will test again to check if sm 
works, would be good to know that suddenly another distribution supports 
nehalem sm.


best regards and thanks again
Cristobal
ps: guess what are the names of the other 2 nodes lol


Acatenango (I said that before), and Pacaya.

Maybe: Santa Maria, Santiaguito, Atitlan, Toliman, San Pedro,
Cerro de Oro ... too many volcanoes, and some are multithreaded ...
You need to buy more nodes!

Gus





On Wed, Jul 28, 2010 at 5:50 PM, Gus Correa > wrote:


Hi Cristobal

Please, read my answer (way down the message) below.

Cristobal Navarro wrote:



On Wed, Jul 28, 2010 at 3:28 PM, Gus Correa
mailto:g...@ldeo.columbia.edu>
>>
wrote:

   Hi Cristobal

   Cristobal Navarro wrote:



   On Wed, Jul 28, 2010 at 11:09 AM, Gus Correa
   mailto:g...@ldeo.columbia.edu>
>
     $which mpirun
/opt/openmpi-1.4.2


  Often times this is a source of confusion, old
versions may
  be first on the PATH.

  Gus


   openMPI version problem is now gone, i can confirm that the
   version is consistent now :), thanks.


   This is good news.


   however, i keep getting this kernel crash randomnly when i
   execute with -np higher than 5
   these are Xeons, with Hyperthreading On, is that a problem??


   The problem may be with Hyperthreading, maybe not.
   Which Xeons?


--> they are not so old, not so new either
fcluster@agua:~$ cat /proc/cpuinfo | more
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU   E5520  @ 2.27GHz
stepping : 5
cpu MHz : 1596.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss h
t tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts
rep_good xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_
cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt
lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 4522.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
...same for cpu1, 2, 3, ..., 15.


AHA! Nehalems!

Here they are E5540, just a different clock speed, I suppose.


information on how the cpu is distributed
fcluster@agua:~$ lstopo
System(7992MB)
 Socket#0 + L3(8192KB)
   L2(256KB) + L1(32KB) + Core#0
 P#0
 P#8
   L2(256KB) + L1(32KB) + Core#1
 P#2
 P#10
   L2(256KB) + L1(32KB) + Core#2
 P#4
 P#12
   L2(256KB) + L1(32KB) + Core#3
 P#6
 P#14
 Socket#1 + L3(8192KB)
   L2(256KB) + L1(32KB) + Core#0
 P#1
 P#9
   L2(256KB) + L1(32KB) + Core#1
 P#3
 P#11
   L2(256KB) + L1(32KB) + Core#2
 P#5
 P#13
   L2(256KB) + L1(32KB) + Core#3
 P#7
 P#15



 
   If I remember right, the old hyperthreading on old Xeons was

   problematic.

   OTOH, about 1-2 months ago I had trouble with OpenMPI on a
   relatively new Xeon Nehalem machine with (the new) Hyperthreading
   turned on,
   and Fedora Core 13.

[OMPI users] MPI broadcast test fails only when I run within a torque job

[Rahul Nabar]

I'm not sure if this is a torque issue or an MPI issue. If I log in to
a compute-node and run the standard mpi broadcast  test it returns no
error but if I run it through PBS/Torque I get an error (see below)
The nodes that return the error are fairly random. Even the same set
of nodes will run a test once and then the next time they fail.  In
case it matters, these nodes have dual interfaces: 1GigE and 10GigE.
All tests I was trying on the same group of 32 nodes.

If I login to the node (just as a regular user ; not as root) then the
test runs fine. No errors at all.

Is there a timeout somewhere? Or some such issue? Not at all sure why
this is happening

Things I've verified. ulimit seems ok. I explicitly have set the
ulimit within the pbs init script as well as in the ssh script that
spawns it.

[root@eu013 ~]# grep ulimit /etc/init.d/pbs
ulimit -l unlimited
[root@eu013 ~]# grep ulimit /etc/init.d/sshd
ulimit -l unlimited


ssh eu013 ulimit -l
unlimited

Even if I put a "ulimit -l" in a PBS job it does return unlimited.

"cat /sys/class/infiniband/cxgb3_0/proto_stats/tcpRetransSegs" returns
a zero on all nodes concerned. Even ifconfig does not return any Error
packets.

-- 
Rahul
#3


PBS command:

mpirun -mca btl openib,sm,self -mca orte_base_help_aggregate 0
/opt/src/mpitests/imb/src/IMB-MPI1 bcast
-through
PBS-
The RDMA CM returned an event error while attempting to make a
connection.  This type of error usually indicates a network
configuration error.

  Local host:   eu013
  Local device: cxgb3_0
  Error name:   RDMA_CM_EVENT_UNREACHABLE
  Peer: eu010

Your MPI job will now abort, sorry.
-
###
Run  physically from a compute node

mpirun -host 
eu001,eu002,eu003,eu004,eu005,eu006,eu007,eu008,eu009,eu010,eu011,eu012,eu013,eu014,eu015,eu016,eu017,eu018,eu019,eu010,eu011,eu012,eu013,eu014,eu015,eu016,eu017,eu018,eu019,eu020,eu021,eu022,eu023,eu024,eu025,eu026,eu027,eu028,eu029,eu030,eu031,eu032
-mca btl openib,sm,self -mca orte_base_help_aggregate 0
/opt/src/mpitests/imb/src/IMB-MPI1 bcast

#
# Benchmarking Bcast
# #processes = 42
#
   #bytes #repetitions  t_min[usec]  t_max[usec]  t_avg[usec]
0 1000 0.02 0.03 0.02
1 1000   170.70   170.76   170.74
2 1000   171.04   171.10   171.08
4 1000   171.09   171.15   171.13
8 1000   171.05   171.13   171.10
   16 1000   171.03   171.10   171.07
   32 100031.9332.0031.98
   64 100028.8629.0228.99
  128 100029.3429.4029.38
  256 100029.9029.9829.95
  512 100030.3930.4730.44
 1024 100031.5931.6731.64
 2048 100038.1538.2638.23
 4096 1000   187.59   187.75   187.68
 8192 1000   208.26   208.41   208.37
16384 1000   395.47   395.71   395.61
32768 1000  9360.99  9441.36  9416.47
65536  400 10522.09 11003.08 10781.73
   131072  299 16971.71 17647.29 17329.27
   262144  160 15404.01 17131.36 15816.46
   524288   80  2659.56  4258.90  3002.04
  1048576   40  4305.72  5305.33  5219.00
  2097152   20  2472.34 10711.80  8599.28
  4194304   10  6275.51 20791.20 13687.10


# All processes entering MPI_Finalize

[OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core

[David Akin]

All,
I'm trying to get the OpenMP portion of the code below to run
multicore on a couple of 8 core nodes.

Good news: Multiple threads are being spawned on each node in the run.
Bad news: Each of the threads only runs on a single core, leaving 7
cores basically idle.
Sorta good news: If I provide a rank file, I get the threads running
on different cores within each node (a hack).

Here's the first lines of output.

 /usr/mpi/gcc/openmpi-1.4/bin/mpirun -host c005,c006 -np 2 -rf
rank.file -x OMP_NUM_THREADS=4 hybrid.gcc

Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
. 
. 
. 

Here's the simple code:
#include 
#include "mpi.h"
#include 

int main(int argc, char *argv[]) {
  int numprocs, rank, namelen;
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int iam = 0, np = 1;
  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
128 */
  int O_ID;/* OpenMP thread ID
 */
  int M_ID;/* MPI rank ID
 */
  int rtn_val;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Get_processor_name(processor_name, &namelen);

  #pragma omp parallel default(shared) private(iam, np,O_ID)
  {
np = omp_get_num_threads();
iam = omp_get_thread_num();
printf("Hello from thread %d out of %d from process %d out of %d on %s\n",
   iam, np, rank, numprocs, processor_name);
int i=0;
int j=0;
double counter=0;
for(i =0;i<;i++)
{
 O_ID = omp_get_thread_num();  /* get OpenMP
thread ID */
 MPI_Get_processor_name(name,&namelen);
 rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
 printf("another parallel region:   name:%s
MPI_RANK_ID=%d OMP_THREAD_ID=%d\n", name,M_ID,O_ID);
 for(j = 0;j<9;j++)
  {
   counter=counter+i;
  }
}

  }

  MPI_Finalize();

}

[OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core

[David Akin]

All,
I'm trying to get the OpenMP portion of the code below to run
multicore on a couple of 8 core nodes.

Good news: multiple threads are being spawned on each node in the run.
Bad news: each of the threads only runs on a single core, leaving 7
cores basically idle.
Sorta good news: if I provide a rank file I get the threads running on
different cores within each node (PITA.

Here's the first lines of output.

 /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2 -rf
rank.file -x OMP_NUM_THREADS=4 hybrid4.gcc

Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
. 
. 
. 

Here's the simple code:
#include 
#include "mpi.h"
#include 

int main(int argc, char *argv[]) {
  int numprocs, rank, namelen;
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int iam = 0, np = 1;
  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
128 */
  int O_ID;/* OpenMP thread ID
 */
  int M_ID;/* MPI rank ID
 */
  int rtn_val;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Get_processor_name(processor_name, &namelen);

  #pragma omp parallel default(shared) private(iam, np,O_ID)
  {
np = omp_get_num_threads();
iam = omp_get_thread_num();
printf("Hello from thread %d out of %d from process %d out of %d on %s\n",
   iam, np, rank, numprocs, processor_name);
int i=0;
int j=0;
double counter=0;
for(i =0;i<;i++)
{
 O_ID = omp_get_thread_num();  /* get OpenMP
thread ID */
 MPI_Get_processor_name(name,&namelen);
 rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
 printf("another parallel region:   name:%s
MPI_RANK_ID=%d OMP_THREAD_ID=%d\n", name,M_ID,O_ID);
 for(j = 0;j<9;j++)
  {
   counter=counter+i;
  }
}

  }

  MPI_Finalize();

}

Re: [OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core

[Ralph Castain]

How are you running it when the threads are all on one core?

If you are specifying --bind-to-core, then of course all the threads will be on 
one core since we bind the process (not the thread). If you are specifying -mca 
mpi_paffinity_alone 1, then the same behavior results.

Generally, if you want to bind threads, the only way to do it is with a rank 
file. We -might- figure out a way to provide an interface for thread-level 
binding, but I'm not sure about that right now. As things stand, OMPI has no 
visibility into the fact that your app spawned threads.


On Jul 28, 2010, at 5:47 PM, David Akin wrote:

> All,
> I'm trying to get the OpenMP portion of the code below to run
> multicore on a couple of 8 core nodes.
> 
> Good news: multiple threads are being spawned on each node in the run.
> Bad news: each of the threads only runs on a single core, leaving 7
> cores basically idle.
> Sorta good news: if I provide a rank file I get the threads running on
> different cores within each node (PITA.
> 
> Here's the first lines of output.
> 
> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2 -rf
> rank.file -x OMP_NUM_THREADS=4 hybrid4.gcc
> 
> Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
> Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
> Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
> Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
> Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
> Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
> Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
> Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
> .
> .
> .
> 
> Here's the simple code:
> #include 
> #include "mpi.h"
> #include 
> 
> int main(int argc, char *argv[]) {
>  int numprocs, rank, namelen;
>  char processor_name[MPI_MAX_PROCESSOR_NAME];
>  int iam = 0, np = 1;
>  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
> 128 */
>  int O_ID;/* OpenMP thread ID
> */
>  int M_ID;/* MPI rank ID
> */
>  int rtn_val;
> 
>  MPI_Init(&argc, &argv);
>  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>  MPI_Get_processor_name(processor_name, &namelen);
> 
>  #pragma omp parallel default(shared) private(iam, np,O_ID)
>  {
>np = omp_get_num_threads();
>iam = omp_get_thread_num();
>printf("Hello from thread %d out of %d from process %d out of %d on %s\n",
>   iam, np, rank, numprocs, processor_name);
>int i=0;
>int j=0;
>double counter=0;
>for(i =0;i<;i++)
>{
> O_ID = omp_get_thread_num();  /* get OpenMP
> thread ID */
> MPI_Get_processor_name(name,&namelen);
> rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
> printf("another parallel region:   name:%s
> MPI_RANK_ID=%d OMP_THREAD_ID=%d\n", name,M_ID,O_ID);
> for(j = 0;j<9;j++)
>  {
>   counter=counter+i;
>  }
>}
> 
>  }
> 
>  MPI_Finalize();
> 
> }
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core

[David Akin]

Here's the exact command I'm running when all threads *are* pinned to
a single core:

/usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2
OMP_NUM_THREADS=4 hybrid4.gcc

Can anyone verify they have the same issue?

On Wed, Jul 28, 2010 at 7:52 PM, Ralph Castain  wrote:
> How are you running it when the threads are all on one core?
>
> If you are specifying --bind-to-core, then of course all the threads will be 
> on one core since we bind the process (not the thread). If you are specifying 
> -mca mpi_paffinity_alone 1, then the same behavior results.
>
> Generally, if you want to bind threads, the only way to do it is with a rank 
> file. We -might- figure out a way to provide an interface for thread-level 
> binding, but I'm not sure about that right now. As things stand, OMPI has no 
> visibility into the fact that your app spawned threads.
>
>
> On Jul 28, 2010, at 5:47 PM, David Akin wrote:
>
>> All,
>> I'm trying to get the OpenMP portion of the code below to run
>> multicore on a couple of 8 core nodes.
>>
>> Good news: multiple threads are being spawned on each node in the run.
>> Bad news: each of the threads only runs on a single core, leaving 7
>> cores basically idle.
>> Sorta good news: if I provide a rank file I get the threads running on
>> different cores within each node (PITA.
>>
>> Here's the first lines of output.
>>
>> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2 -rf
>> rank.file -x OMP_NUM_THREADS=4 hybrid4.gcc
>>
>> Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
>> Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>> Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>> Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
>> Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
>> Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>> Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>> Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>> another parallel region:       name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>> another parallel region:       name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>> .
>> .
>> .
>>
>> Here's the simple code:
>> #include 
>> #include "mpi.h"
>> #include 
>>
>> int main(int argc, char *argv[]) {
>>  int numprocs, rank, namelen;
>>  char processor_name[MPI_MAX_PROCESSOR_NAME];
>>  int iam = 0, np = 1;
>>  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
>> 128         */
>>  int O_ID;                            /* OpenMP thread ID
>>         */
>>  int M_ID;                            /* MPI rank ID
>>         */
>>  int rtn_val;
>>
>>  MPI_Init(&argc, &argv);
>>  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>>  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>  MPI_Get_processor_name(processor_name, &namelen);
>>
>>  #pragma omp parallel default(shared) private(iam, np,O_ID)
>>  {
>>    np = omp_get_num_threads();
>>    iam = omp_get_thread_num();
>>    printf("Hello from thread %d out of %d from process %d out of %d on %s\n",
>>           iam, np, rank, numprocs, processor_name);
>>    int i=0;
>>    int j=0;
>>    double counter=0;
>>    for(i =0;i<;i++)
>>            {
>>             O_ID = omp_get_thread_num();          /* get OpenMP
>> thread ID                 */
>>             MPI_Get_processor_name(name,&namelen);
>>             rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
>>             printf("another parallel region:       name:%s
>> MPI_RANK_ID=%d OMP_THREAD_ID=%d\n", name,M_ID,O_ID);
>>             for(j = 0;j<9;j++)
>>              {
>>               counter=counter+i;
>>              }
>>            }
>>
>>  }
>>
>>  MPI_Finalize();
>>
>> }
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/l

Re: [OMPI users] Hybrid OpenMPI / OpenMP run pins OpenMP threads to a single core

[Ralph Castain]

Something doesn't add up - the default for ompi is to -not- bind. Check your 
default mca param file and your environment. Do you have any mca params set in 
them?


On Jul 28, 2010, at 9:40 PM, David Akin wrote:

> Here's the exact command I'm running when all threads *are* pinned to
> a single core:
> 
> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2
> OMP_NUM_THREADS=4 hybrid4.gcc
> 
> Can anyone verify they have the same issue?
> 
> On Wed, Jul 28, 2010 at 7:52 PM, Ralph Castain  wrote:
>> How are you running it when the threads are all on one core?
>> 
>> If you are specifying --bind-to-core, then of course all the threads will be 
>> on one core since we bind the process (not the thread). If you are 
>> specifying -mca mpi_paffinity_alone 1, then the same behavior results.
>> 
>> Generally, if you want to bind threads, the only way to do it is with a rank 
>> file. We -might- figure out a way to provide an interface for thread-level 
>> binding, but I'm not sure about that right now. As things stand, OMPI has no 
>> visibility into the fact that your app spawned threads.
>> 
>> 
>> On Jul 28, 2010, at 5:47 PM, David Akin wrote:
>> 
>>> All,
>>> I'm trying to get the OpenMP portion of the code below to run
>>> multicore on a couple of 8 core nodes.
>>> 
>>> Good news: multiple threads are being spawned on each node in the run.
>>> Bad news: each of the threads only runs on a single core, leaving 7
>>> cores basically idle.
>>> Sorta good news: if I provide a rank file I get the threads running on
>>> different cores within each node (PITA.
>>> 
>>> Here's the first lines of output.
>>> 
>>> /usr/mpi/gcc/openmpi-1.4-qlc/bin/mpirun -host c005,c006 -np 2 -rf
>>> rank.file -x OMP_NUM_THREADS=4 hybrid4.gcc
>>> 
>>> Hello from thread 2 out of 4 from process 1 out of 2 on c006.local
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=2
>>> Hello from thread 3 out of 4 from process 1 out of 2 on c006.local
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>>> Hello from thread 1 out of 4 from process 1 out of 2 on c006.local
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>>> Hello from thread 1 out of 4 from process 0 out of 2 on c005.local
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=1
>>> Hello from thread 3 out of 4 from process 0 out of 2 on c005.local
>>> Hello from thread 2 out of 4 from process 0 out of 2 on c005.local
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>> Hello from thread 0 out of 4 from process 0 out of 2 on c005.local
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>>> Hello from thread 0 out of 4 from process 1 out of 2 on c006.local
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=0
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=3
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=3
>>> another parallel region:   name:c005.local MPI_RANK_ID=0 OMP_THREAD_ID=2
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=0
>>> another parallel region:   name:c006.local MPI_RANK_ID=1 OMP_THREAD_ID=1
>>> .
>>> .
>>> .
>>> 
>>> Here's the simple code:
>>> #include 
>>> #include "mpi.h"
>>> #include 
>>> 
>>> int main(int argc, char *argv[]) {
>>>  int numprocs, rank, namelen;
>>>  char processor_name[MPI_MAX_PROCESSOR_NAME];
>>>  int iam = 0, np = 1;
>>>  char name[MPI_MAX_PROCESSOR_NAME];   /* MPI_MAX_PROCESSOR_NAME ==
>>> 128 */
>>>  int O_ID;/* OpenMP thread ID
>>> */
>>>  int M_ID;/* MPI rank ID
>>> */
>>>  int rtn_val;
>>> 
>>>  MPI_Init(&argc, &argv);
>>>  MPI_Comm_size(MPI_COMM_WORLD, &numprocs);
>>>  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
>>>  MPI_Get_processor_name(processor_name, &namelen);
>>> 
>>>  #pragma omp parallel default(shared) private(iam, np,O_ID)
>>>  {
>>>np = omp_get_num_threads();
>>>iam = omp_get_thread_num();
>>>printf("Hello from thread %d out of %d from process %d out of %d on 
>>> %s\n",
>>>   iam, np, rank, numprocs, processor_name);
>>>int i=0;
>>>int j=0;
>>>double counter=0;
>>>for(i =0;i<;i++)
>>>{
>>> O_ID = omp_get_thread_num();  /* get OpenMP
>>> thread ID */
>>> MPI_Get_processor_name(name,&namelen);
>>> rtn_val = MPI_Comm_rank(MPI_COMM_WORLD,&M_ID);
>>> printf("another parallel region:   name:%s
>>> MPI_RANK