Re: [OMPI users] Running application with MPI_Comm_spawn() in multithreaded environment

Ralph Castain Mon, 6 Oct 2008 09:26:35 -0400

Hi Roberto

My time is somewhat limited, so I couldn't review the code in detail.However, I think I got the gist of it.


A few observations:

1. the code is rather inefficient, if all you want to do is spawn apattern of slave processes based on a file. Unless there is someoverriding reason for doing this one comm_spawn at a time, it would befar faster to issue a single comm_spawn and just provide the hostfileto us. You could use either the seq or rank_file mapper - both wouldtake the file and provide the outcome you seek. The only differencewould be that the child procs would all be in the same comm_world -don't know if that is an issue or not.

2. OMPI definitely cannot handle the threaded version of this code atthis time - not sure when we will get to it.

3. if you serialize the code, we -should- be able to handle it.However, I'm not entirely sure your current method actually does that.It looks like you call comm_spawn, and then create a new thread whichthen calls comm_spawn. I'm afraid I can't quite figure out how thethread locking would occur to prevent multiple threads continuing tocall comm_spawn - you might want to check it again and ensure it iscorrect. Frankly, I'm not entirely sure what the thread creation isgaining you - as I said, we can only call comm_spawn serially, sohaving multiple threads would seem to be unnecessary...unless thiscode is incomplete and you need the threads for some other purpose.

Again, you might look at that loop_spawn code I mentioned before tosee a working example. Alternatively, if your code works under HP MPI,you might want to stick with it for now until we get the threadingsupport up to your required level.


Hope that helps
Ralph

On Oct 3, 2008, at 10:36 AM, Roberto Fichera wrote:

Ralph Castain ha scritto:

Interesting. I ran a loop calling comm_spawn 1000 times without a
problem. I suspect it is the threading that is causing the troublehere.

I think so! My guessing is that at low level there is some troublewhen

handling *concurrent*
orted spawning. Maybe

You are welcome to send me the code. You can find my loop code inyour
code distribution under orte/test/mpi - look for loop_spawn and
loop_child.

In the attached code the spawing logic is currently under a loop inthe

main of the testmaster, so it's completly
unthreaded at least until the MPI_Comm_spawn() terminate its work. If
you wish like to test multithreading spawing
you can comment the NodeThread_spawnSlave() in the main loop and
uncomment the same function in the

NodeThread_threadMain(). Finally if you want multithreading spawningbut

serialized against a mutex than uncomment
the pthread_mutex_lock/unlock() in the NodeThread_threadMain().

This code run *without* any trouble in the HP MPI implementation. It
works not so well in mpich2 trunk version due
to two problems: limit of ~24.4K context id and/or a race in poll()
while waiting a termination under MPI_Comm_disconnect()
concurrently with a MPI_Comm_spawn().


Ralph

On Oct 3, 2008, at 9:11 AM, Roberto Fichera wrote:

Ralph Castain ha scritto:


On Oct 3, 2008, at 7:14 AM, Roberto Fichera wrote:

Ralph Castain ha scritto:
I committed something to the trunk yesterday. Given thecomplexity of
the fix, I don't plan to bring it over to the 1.3 branch until
sometime mid-to-end next week so it can be adequately tested.
Ok! So it means that I can checkout from the SVN/trunk to getyou fix,
right?


Yes, though note that I don't claim it is fully correct yet. Still
needs testing. However, I have tested it a fair amount and it seems
okay.

If you do test it, please let me know how it goes.

I execute my test on the svn/trunk below

              Open MPI: 1.4a1r19677
 Open MPI SVN revision: r19677
 Open MPI release date: Unreleased developer copy
              Open RTE: 1.4a1r19677
 Open RTE SVN revision: r19677
 Open RTE release date: Unreleased developer copy
                  OPAL: 1.4a1r19677
     OPAL SVN revision: r19677
     OPAL release date: Unreleased developer copy
          Ident string: 1.4a1r19677

below is the output which seems to freeze just after the secondspawn.


[roberto@master TestOpenMPI]$ mpirun --verbose --debug-daemons
--hostfile $PBS_NODEFILE -wdir "`pwd`" -np 1 testmaster 100000
$PBS_NODEFILE
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
add_local_procs

[master.tekno-soft.it:30063] [[19516,0],0] node[0].name masterdaemon 0

arch ffc91200

[master.tekno-soft.it:30063] [[19516,0],0] node[1].name cluster4daemon

INVALID arch ffc91200

[master.tekno-soft.it:30063] [[19516,0],0] node[2].name cluster3daemon

INVALID arch ffc91200

[master.tekno-soft.it:30063] [[19516,0],0] node[3].name cluster2daemon

INVALID arch ffc91200

[master.tekno-soft.it:30063] [[19516,0],0] node[4].name cluster1daemon

INVALID arch ffc91200
Initializing MPI ...
[master.tekno-soft.it:30063] [[19516,0],0] orted_recv: received
sync+nidmap from local proc [[19516,1],0]
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
collective data cmd
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
message_local_procs
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
collective data cmd
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
message_local_procs
Loading the node's ring from file
'/var/torque/aux//932.master.tekno-soft.it'
... adding node #1 host is 'cluster4.tekno-soft.it'
... adding node #2 host is 'cluster3.tekno-soft.it'
... adding node #3 host is 'cluster2.tekno-soft.it'
... adding node #4 host is 'cluster1.tekno-soft.it'
A 4 node's ring has been made
At least one node is available, let's start to distribute 100000 job
across 4 nodes!!!
Setting up the host as 'cluster4.tekno-soft.it'
Setting the work directory as '/data/roberto/MPI/TestOpenMPI'
Spawning a task 'testslave.sh' on node 'cluster4.tekno-soft.it'

Daemon was launched on cluster4.tekno-soft.it - beginning toinitialize

Daemon [[19516,0],1] checking in as pid 25123 on host
cluster4.tekno-soft.it
Daemon [[19516,0],1] not using static ports
[cluster4.tekno-soft.it:25123] [[19516,0],1] orted: up and running -
waiting for commands!
[master.tekno-soft.it:30063] [[19516,0],0] orted_cmd: received
add_local_procs