Re: [OMPI users] dimension mapping of MPI_Cart_create

[Paul Hilscher]

Dear Jeff, thanks for the information.

>Open MPI currently has very limited cartesian support -- it actually
doesn't remap anything.

I see, OpenMPI doesn't remap anything; this explains probably why my runtime
of my simulation varies sometimes between 30% for the same setup.

>Would you have any interest in writing a partitioning algorithm for your
needs within the context of a plugin?  I'd be happy to walk >you through the
process; it's not too complicated (although we should probably move the
discussion off to the Open MPI devel >mailing list).

I guess after using for more than a decade Open Source Software, it's time
to give something back :). ... so yes, I am willing to do that !!

Because I am not yet experienced with OpenMPI internals, I would really
appreciate your advice, if you could tell me where exactly I have to dig
into.. I guess it should be around ompi_topo_create function, but how to
write MPI_Cart_Create as a plugin, I will rely on you information. And do
you know if MPICH, LAM etc. have an efficient implementation of
MPI_Cart_Create ? so I can borrow some ideas from them

best wishes,

Paul Hilscher



On Tue, Jun 29, 2010 at 8:17 PM, Jeff Squyres  wrote:

> Open MPI currently has very limited cartesian support -- it actually
> doesn't remap anything.
>
> That being said, it is *very* easy to extend Open MPI's algorithms for
> cartesian partitioning.  As you probably already know, Open MPI is all about
> its plugins -- finding and selecting a good set of plugins to use at
> run-time.  Open MPI has many different types of plugins.  One of these types
> of plugins performs the cartesian/graph mapping behind MPI_Cart_create (and
> friends) function(s).
>
> Would you have any interest in writing a partitioning algorithm for your
> needs within the context of a plugin?  I'd be happy to walk you through the
> process; it's not too complicated (although we should probably move the
> discussion off to the Open MPI devel mailing list).
>
>
> On Jun 29, 2010, at 4:50 AM, Paul Hilscher wrote:
>
> > Dear OpenMPI list,
> >
> > I am using  a MPI-parallelized simulation program,  with a
> domain-decomposition in 6-Dimensions.
> > In order to improve the scalability of my program I would like to know
> according to what preferences
> > is MPI distributing the ranks when using MPI_Cart_create( reorder
> allowed).
> >
> > To explain my inquiry, imagine a 3-dimensional solver in  X-Y-M and 4
> computing
> > nodes, each nodes consist of 4 Quad-Core CPUs (4(Node)x[ 4(CPUs) x
> 4(Cores))] CPUs=64CPUs).
> >
> > Now I decompose  all 3 dimensions by 4 (4x4x4 = 64) using
>  MPI_Cart_create.
> > MPI has now several  possibilities to map the problem e.g. X-M (locally)
> on a node and
> > Y across the nodes, or Y-M (locally) and X across the nodes.
> >
> > Now my question is, how can I tell MPI that I want to distribute X-Y
> locally while
> > M is distributed across nodes. The reason is that X-Y
> > communication ratio is much large (FFT) compared to M where we have only
> > 2 communications per time-step via an Allreduce.
> > An MPI implementation for the BlueGENE for example has an option
> > called mapfile where on can tell MPI how to map the dimensions onto
> > the Nodes. I did not found somethings similar for openmpi.
> >
> > Does anybody know how to achieve this mapping or could anybody
> > tell me where I could find some examples or tutorials ?
> >
> > Thank you very much for your help and best wishes
> >
> > Paul Hilscher
> > ___
> >
>

Re: [OMPI users] about MPI

[Jeff Squyres]

On Jun 29, 2010, at 9:35 PM, 王睿 wrote:

> Thanks for the feedback. More below:
> 
> Is there any MPI implementions which meet the following requirements:
> 
> 1, it doesn't terminate the whole job when a node is dead?
> 
> 2, it allows the spare node to replace the dead node and take over the work 
> of the dead node?
> 
> As far as I know, FT-MPI meets the two requirements, but it hasn't updated 
> since 2004. Open-mpi is said to combine serveral projects including FT-MPI, 
> but so far, it only provides checkpoinr/restart as a way of fault-tolerance. 

I know that the UT people have been working on such things over the past few 
years, but I don't know the current status.

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

Re: [OMPI users] MPI Persistent Communication Question

[amjad ali]

and it's conceivable that you might have better performance with
>
> CALL MPI_ISEND()
> DO I = 1, N
> call do_a_little_of_my_work()  ! no MPI progress is being made here
> CALL MPI_TEST()! enough MPI progress is being made here
> that the receiver has something to do
> END DO
> CALL MPI_WAIT()
>
> Whether performance improves or not is not guaranteed by the MPI standard.
>
> And the SECOND desire is to use Persistent communication for even better
> speedup.
>
> Right.  That's a separate issue.
>
>

So actually I am focusing on the persistent communication at this time.
Based on your suggestions, I developed:

sending, receiving buffers, and the request array is defined in declared in
the global module. And their sizes are allocated in the main program. But
following is not working. Segmentation fault messages at just from the
underline blue line lace.

*Main program starts--@@@.*
*
**CALL MPI_RECV_INIT for each neighboring process  **
CALL MPI_SEND_INIT for each neighboring process*
*Loop Calling the subroutine1(1 times in the main
program).

** Call subroutine1*
*
**Subroutine1 starts===*
*
   Loop A starts here  (three passes)
   Call subroutine2

   Subroutine2 starts

 Pick local data from array U in separate arrays for each
neighboring processor
 CALL MPI_STARTALL
 ---perform work that could be done with local data
 CALL MPI_WAITALL( )
 ---perform work using the received data
   Subroutine**2** ends***


* ---perform work to update array U*
*   Loop A ends here *
*Subroutine1 ends*

*Loop Calling the subroutine1 ends(1 times in the main
program).*

*CALL MPI_Request_free( )*

*Main program ends--@@@.*

How to tackle all this.

Re: [OMPI users] MPI Persistent Communication Question

[Eugene Loh]

amjad ali wrote:

  
  
and it's conceivable that you
might have better performance with

    CALL MPI_ISEND()
    DO I = 1, N
    call do_a_little_of_my_work()  ! no MPI progress is being made
here
    CALL MPI_TEST()    ! enough MPI progress is being made
here that the receiver has something to do
    END DO
    CALL MPI_WAIT()

Whether performance improves or not is not guaranteed by the MPI
standard.

And the SECOND desire is to use Persistent communication
for even better speedup.


Right.  That's a separate issue.
  
  
So actually I am focusing on the persistent communication at this time.
Based on your suggestions, I developed:
  
  
  
sending, receiving buffers, and the request array is defined in
declared in the global module. And their sizes are allocated in the
main program. But following is not working. Segmentation fault messages
at just from the underline blue line lace.

Well, the problem must be in the details of how you're implementing
this.  I've attached a program that works for me.

  Main program
starts--@@@.
  
  CALL MPI_RECV_INIT for
each neighboring process  
CALL MPI_SEND_INIT for each neighboring process
  Loop Calling the
subroutine1(1 times in the main program).
  
   Call subroutine1
  
  Subroutine1
starts===
  
     Loop A starts here

(three passes)
  
   Call subroutine2
  
   Subroutine2 starts
  
         Pick local data from
array U in separate arrays for each neighboring processor
  
           CALL MPI_STARTALL
           ---perform work that could
be done with local data
       CALL MPI_WAITALL( )
       ---perform work
using the received data
   Subroutine2 ends
  
  
  
---perform work to update array U
     Loop A ends here

  
  Subroutine1
ends
  
  Loop Calling the subroutine1
ends(1 times in the main program).
  
  CALL MPI_Request_free( )
  
  Main program
ends--@@@.
  
How to tackle all this.



module my_mpi_stuff
  integer, parameter :: nmsgs = 1, nwords = 8
  integer me, np
  integer reqs(   nmsgs,2)   ! (...,1) are for sends and (...,2) are 
for receives
  real(8) bufs(nwords,nmsgs,2)   ! (...,1) are for sends and (...,2) are 
for receives
end module my_mpi_stuff

program main
  use my_mpi_stuff
  include "mpif.h"

  call MPI_Init(ier)
  call MPI_Comm_size(MPI_COMM_WORLD,np,ier)
  call MPI_Comm_rank(MPI_COMM_WORLD,me,ier)

  ! set up individual sends and receives
  if ( np /= 2 ) stop "np is not 2"  ! this simple example works only for 
np==2
  call MPI_Recv_init(bufs(1,1,2), nwords, MPI_REAL8, 1-me, 300, MPI_COMM_WORLD, 
reqs(1,2), ier)
  call MPI_Send_init(bufs(1,1,1), nwords, MPI_REAL8, 1-me, 300, MPI_COMM_WORLD, 
reqs(1,1), ier)

  do i = 1, 1
call sub1()
  end do

  ! dump out buffers
  do imsg = 1, nmsgs
write(6,'(3i5,8f8.1)') me, imsg, 1, bufs(:,imsg,1)
write(6,'(3i5,8f8.1)') me, imsg, 2, bufs(:,imsg,2)
  end do

  do imsg = 1, nmsgs
call MPI_Request_free(reqs(imsg,1), ier)
call MPI_Request_free(reqs(imsg,2), ier)
  end do

  call MPI_Finalize(ier)
end program main

subroutine sub1()
  do i = 1, 3
call sub2()
! call update(u)
  end do
end subroutine sub1

subroutine sub2()
  use my_mpi_stuff
  include "mpif.h"

  ! Pick local data from array U in separate arrays for each neighboring 
processor
  do imsg = 1, nmsgs
do iword = 1, nwords
  bufs(iword,imsg,1) = 1 * me + 100 * imsg + iword
end do
  end do

  call MPI_Startall(2*nmsgs,reqs,ier)

  ! ---perform work that could be done with local data

  call MPI_Waitall (2*nmsgs,reqs,MPI_STATUSES_IGNORE,ier)

  ! ---perform work using the received data

end subroutine sub2

#!/bin/csh

setenv OPAL_PREFIX .
set path = ( $OPAL_PREFIX/bin $path )

mpif90 a.f90
mpirun -n 2 ./a.out

[OMPI users] Open MPI, Segmentation fault

[Jack Bryan]

Dear All,
I am using Open MPI, I got the error: 
n337:37664] *** Process received signal ***[n337:37664] Signal: Segmentation 
fault (11)[n337:37664] Signal code: Address not mapped (1)[n337:37664] Failing 
at address: 0x7fffcfe9[n337:37664] [ 0] /lib64/libpthread.so.0 
[0x3c50e0e4c0][n337:37664] [ 1] 
/lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2 
[0x414ed7][n337:37664] [ 2] /lib64/libc.so.6(__libc_start_main+0xf4) 
[0x3c5021d974][n337:37664] [ 3] 
/lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2(__gxx_personality_v0+0x1f1)
 [0x412139][n337:37664] *** End of error message ***
After searching answers, it seems that some functions fail.  My program can run 
well for 1,2,10 processors, but fail when the number of tasks cannotbe divided 
evenly by number of processes. 
Any help is appreciated. 
thanks
Jack
June 30  2010
  
_
The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with 
Hotmail. 
http://www.windowslive.com/campaign/thenewbusy?tile=multicalendar&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_5

Re: [OMPI users] Open MPI, Segmentation fault

[David Zhang]

When I got segmentation faults, it has always been my coding mistakes.
Perhaps your code is not robust against number of processes not divisible by
2?

On Wed, Jun 30, 2010 at 8:47 AM, Jack Bryan  wrote:

>  Dear All,
>
> I am using Open MPI, I got the error:
>
> n337:37664] *** Process received signal ***
> [n337:37664] Signal: Segmentation fault (11)
> [n337:37664] Signal code: Address not mapped (1)
> [n337:37664] Failing at address: 0x7fffcfe9
> [n337:37664] [ 0] /lib64/libpthread.so.0 [0x3c50e0e4c0]
> [n337:37664] [ 1]
> /lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2 [0x414ed7]
> [n337:37664] [ 2] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c5021d974]
> [n337:37664] [ 3]
> /lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2(__gxx_personality_v0+0x1f1)
> [0x412139]
> [n337:37664] *** End of error message ***
>
> After searching answers, it seems that some functions fail.
>
> My program can run well for 1,2,10 processors, but fail when the number of
> tasks cannot
> be divided evenly by number of processes.
>
> Any help is appreciated.
>
> thanks
>
> Jack
>
> June 30  2010
>
>
> --
> The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with
> Hotmail. Get 
> busy.
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
David Zhang
University of California, San Diego

Re: [OMPI users] MPI Persistent Communication Question

[amjad ali]

Dear E. loh,

Thank u very much for your help.

Actually i was doing the same according to your earlier suggestions
---and now in the program; but error was there.
At last i found the blunder made by myself.
It was a typo mistake infact of a variable name.

i will let u know about the performance comparison.

Thanks again.

Best regards
AA

Re: [OMPI users] Open MPI, Segmentation fault

[amjad ali]

Based on my experiences, I would FULLY endorse (100% agree with) David
Zhang.
It is usually a coding or typo mistake.

At first, Ensure that array sizes and dimension are correct.

I experience that if openmpi is compiled with gnu compilers (not with Intel)
then it also point outs the subroutine exactly in which the fault occur.
have a try.

best,
AA



On Wed, Jun 30, 2010 at 12:43 PM, David Zhang  wrote:

> When I got segmentation faults, it has always been my coding mistakes.
> Perhaps your code is not robust against number of processes not divisible by
> 2?
>
> On Wed, Jun 30, 2010 at 8:47 AM, Jack Bryan wrote:
>
>>  Dear All,
>>
>> I am using Open MPI, I got the error:
>>
>> n337:37664] *** Process received signal ***
>> [n337:37664] Signal: Segmentation fault (11)
>> [n337:37664] Signal code: Address not mapped (1)
>> [n337:37664] Failing at address: 0x7fffcfe9
>> [n337:37664] [ 0] /lib64/libpthread.so.0 [0x3c50e0e4c0]
>> [n337:37664] [ 1]
>> /lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2 [0x414ed7]
>> [n337:37664] [ 2] /lib64/libc.so.6(__libc_start_main+0xf4) [0x3c5021d974]
>> [n337:37664] [ 3]
>> /lustre/home/rhascheduler/RhaScheduler-0.4.1.1/mytest/nmn2(__gxx_personality_v0+0x1f1)
>> [0x412139]
>> [n337:37664] *** End of error message ***
>>
>> After searching answers, it seems that some functions fail.
>>
>> My program can run well for 1,2,10 processors, but fail when the number of
>> tasks cannot
>> be divided evenly by number of processes.
>>
>> Any help is appreciated.
>>
>> thanks
>>
>> Jack
>>
>> June 30  2010
>>
>>
>> --
>> The New Busy think 9 to 5 is a cute idea. Combine multiple calendars with
>> Hotmail. Get 
>> busy.
>>
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>
>
>
>
> --
> David Zhang
> University of California, San Diego
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>

[OMPI users] what is "thread support: progress" ?

[Riccardo Murri]

Hello,

I just re-compiled OMPI, and noticed this in the
"ompi_info --all" output:

Open MPI: 1.4.3a1r23323
...
  Thread support: posix (mpi: yes, progress: no)
...

what is this "progress thread support"?  Is it the "asynchronous
progress ... in the TCP point-to-point device" that the FAQ
mentions?  I could not find any ./configure option to enable or
disable it.

Cheers,
Riccardo

Re: [OMPI users] what is "thread support: progress" ?

[Ralph Castain]

A stale, unsupported option - we have removed it in future releases to remove 
confusion.

On Jun 30, 2010, at 2:16 PM, Riccardo Murri wrote:

> Hello,
> 
> I just re-compiled OMPI, and noticed this in the
> "ompi_info --all" output:
> 
>Open MPI: 1.4.3a1r23323
>...
>  Thread support: posix (mpi: yes, progress: no)
>...
> 
> what is this "progress thread support"?  Is it the "asynchronous
> progress ... in the TCP point-to-point device" that the FAQ
> mentions?  I could not find any ./configure option to enable or
> disable it.
> 
> Cheers,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] is OpenMPI 1.4 thread-safe?

[Riccardo Murri]

Hello,

The FAQ states: "Support for MPI_THREAD_MULTIPLE [...] has been
designed into Open MPI from its first planning meetings.  Support for
MPI_THREAD_MULTIPLE is included in the first version of Open MPI, but
it is only lightly tested and likely still has some bugs."

The man page of "mpirun" from v1.4.3a1r23323 in addition says "Open
MPI is, currently, neither thread-safe nor async-signal-safe" (section
"Process Termination / Signal Handling").

Are these statements up-to-date?  What is the status of
MPI_THREAD_MULTIPLE in OMPI 1.4?

Thanks in advance for any info!

Cheers,
Riccardo

[OMPI users] Parallel Tempering with MPI

[Asad Ali]

Hi all,

I am working on a parallel tempering MCMC code using OpenMPI scripts. I am a
bit confused about proposing swaps between chains running on different
cores.
I know how to propose swaps but I am not sure as to where to to do it (i.e.
how to specify an independent node or core for it.). If some body is/was
working on parallel tempering MCMC using MPI then please help me. An example
code would be really helpful.

Cheers,

Asad

On Thu, Jul 1, 2010 at 9:28 AM, Riccardo Murri wrote:

> Hello,
>
> The FAQ states: "Support for MPI_THREAD_MULTIPLE [...] has been
> designed into Open MPI from its first planning meetings.  Support for
> MPI_THREAD_MULTIPLE is included in the first version of Open MPI, but
> it is only lightly tested and likely still has some bugs."
>
> The man page of "mpirun" from v1.4.3a1r23323 in addition says "Open
> MPI is, currently, neither thread-safe nor async-signal-safe" (section
> "Process Termination / Signal Handling").
>
> Are these statements up-to-date?  What is the status of
> MPI_THREAD_MULTIPLE in OMPI 1.4?
>
> Thanks in advance for any info!
>
> Cheers,
> Riccardo
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users
>



-- 
"Statistical thinking will one day be as necessary for efficient citizenship
as the ability to read and write." - H.G. Wells