Re: [OMPI users] Using Boost::Thread for multithreading within OpenMPI processes

[Nick Edmonds]

Hi Jacky,

I'm a regular reader of this list but seldom a poster.  In this case however I 
might actually be qualified to answer some questions or provide some insight 
given I'm not sure how many other folks here use Boost.Thread.  The first 
question is really what sort of threading model you want to use with MPI, which 
others here are probably more qualified to advise you on.  

In our applications we're using Boost.Thread with MPI_THREAD_MULTIPLE, which is 
a not all-together enjoyable experience because the openib BTL lacks support 
for thread multiple (at least as of the last time I checked).  That being said, 
Boost.Thread behaves just like any pthread code on the linux clusters we run 
on, as well as one BlueGene/P.  With MPI_THREAD_SERIALIZED writing 
hybrid-parallel code is pretty painless.  Most of the work required involved 
adding two-stage collectives such that threads first perform collectives 
locally and then a single thread participates in the MPI collective operation.  

If you end up using Boost.MPI you could probably even write your own wrappers 
to encapsulate the local computation required for MPI collective operations.  
Unfortunately Boost.MPI currently lacks full support for even MPI-2 but if it 
includes the subset of functionality you need it may be worthwhile.  Extensions 
are fairly straightforward to implement as well.

I've implemented a few different approaches to MPI + threading in the context 
of Boost, from explicit thread management to thread pools, and currently a 
complete runtime system.  Most of it is research code, though there's no reason 
it couldn't be released, and some of it probably will be eventually.  If you'd 
like to describe your intended use case I'm happy to offer any advice I can 
based on what I've learned.

Cheers,
Nick

On Apr 22, 2013, at 3:25 PM, Thomas Watson wrote:

> Hi,
> 
> I would like to create a pool of threads (using Boost::Thread) within each 
> OpenMPI process to accelerate my application on multicore CPUs. My 
> application is already built on OpenMPI, but it currently exploits 
> parallelism only at the process level. 
> 
> I am wondering if anyone can point me to some good 
> tutorials/documents/examples on how to integrate Boost multithreading with 
> OpenMPI applications?
> 
> Thanks!
> 
> Jacky
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

[OMPI users] openib BTL and MPI_THREAD_MULTIPLE

[Nick Edmonds]

Currently the openib BTL silently refuses to run when MPI_THREAD_MULTIPLE is 
specified (ompi/mca/btl/openib/btl_openib_component.c:2367 in the current 
trunk, r22822) which leads to confusing (to some people) error messages such as:

PML add procs failed  --> Returned "Unreachable" (-12) instead of "Success" (0)

Would it be possible to provide a warning/error indicating that the BTL failed 
to load, and why?  The logical next question would be, is anyone working on an 
openib BTL that supports MPI_THREAD_MULTIPLE?  I'm currently stuck using IPoIB 
which is obviously undesirable from a performance standpoint.

Thanks in advance for any information anyone can provide.

Cheers,
Nick