On 08/16/2011 02:11 PM, Ralph Castain wrote:
That should work, then. When you set the "host" property, did you give the same
name as was in your machine file?
Debug options that might help:
-mca plm_base_verbose 5 -mca rmaps_base_verbose 5
You'll need to configure --enable-debug to get the output, but that should help
tell us what is happening.
To be clear here is the code I am using to spawn the MPI job:
// create the info object
MPI_Info info;
MPI_Info_create(&info);
MPI_Info_set(info, "host", const_cast<char*>(hostname.c_str()));
LOG(ERROR)<< hostname;
LOG(DEBUG)<< "Invoking task ID '"<< task_id<<"': '"<< exec_name<< "'";
MPI_Comm_spawn( const_cast<char*>(exec_name.c_str()), cargs, num_procs,
info, 0, MPI_COMM_SELF,&intercomm, MPI_ERRCODES_IGNORE );
delete[] cargs;
MPI_Info_free(&info);
and here is the log message:
In this case the MPI_Spaw creates a job with 3 MPI processes. As you can see MPI_Spawn
doesn't care about my "host" setting, it just goes ahead and map the processes
to node b05 and node b06 which are in my machinefile. (which is the same as before)
is there any way to overwrite this behaviour?
DEBUG 14628:R<0> 17:00:13] Spawning new MPI processes...
DEBUG 14628:R<0> 17:00:13] Serving event 'TASK_CREATED', (number of registered
handlers: 1)
ERROR 14628:R<0> 17:00:13] b01
DEBUG 14628:R<0> 17:00:13] Invoking task ID '4': './simulator'
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive got message
from [[34621,1],0]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive job launch
command
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:rsh: setting up job
[34621,4]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:setup_job for job
[34621,4]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: created
new proc [[34621,4],INVALID]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot mapping
proc in job [34621,4] to node b02
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: adding node b02 to
map
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for
job [34621,4] to node b02
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: created
new proc [[34621,4],INVALID]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot mapping
proc in job [34621,4] to node b01
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: adding node b01 to
map
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for
job [34621,4] to node b01
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot: created
new proc [[34621,4],INVALID]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:claim_slot mapping
proc in job [34621,4] to node b02
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base: mapping proc for
job [34621,4] to node b02
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:compute_usage
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons
existing daemon [[34621,0],2] already launched
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] rmaps:base:define_daemons
existing daemon [[34621,0],1] already launched
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:rsh: no new daemons to
launch
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch_apps for job
[34621,4]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:report_launched for
job [34621,4]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
from daemon [[34621,0],0]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
completed processing
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
reissuing non-blocking recv
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
from daemon [[34621,0],1]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched
for proc [[34621,4],1] from daemon [[34621,0],1]: pid 14646 state 2 exit 0
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
completed processing
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
reissuing non-blocking recv
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
from daemon [[34621,0],2]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched
for proc [[34621,4],0] from daemon [[34621,0],2]: pid 9803 state 2 exit 0
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launched
for proc [[34621,4],2] from daemon [[34621,0],2]: pid 9804 state 2 exit 0
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:app_report_launch
completed processing
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:report_launched all
apps reported
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch wiring up iof
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:launch completed for
job [34621,4]
[kreusspitze.dps.uibk.ac.at:02647] [[34621,0],0] plm:base:receive job [34621,4]
launched
cheers, Simone P.
On Aug 16, 2011, at 5:09 AM, Simone Pellegrini wrote:
On 08/16/2011 12:30 PM, Ralph Castain wrote:
What version are you using?
OpenMPI 1.4.3
On Aug 16, 2011, at 3:19 AM, Simone Pellegrini wrote:
Dear all,
I am developing a system to manage MPI tasks on top of MPI. The architecture is
rather simple, I have a set of scheduler processes which takes care to manage
the resources of a node. The idea is to have 1 (or more) of those scheduler
allocated on each node of a cluster and then create new MPI processes (on
demand) as computation is needed. Allocation of processes is done using
MPI_Spawn.
The system now works fine on a single node by allocating the main scheduler
using the following mpi command:
mpirun --np 1 ./scheduler ...
Now when I scale to multiple nodes problems with default MPI behaviour starts.
For example lets assume I have 2 nodes with 8 cpu cores each. I therefore set
up a machine file in the following way:
s01 slots=1
s02 slots=1
and start the node schedulers in the following way:
mpirun --np 2 --machinefile machinefile ./scheduler ...
This allocates the processes correctly, now the problem starts when I invoke
MPI_Spawn. basically MPI spawn also uses the informations from the machinefile
and if 4 MPI processes are spawned 2 are allocated in s01 and 2 on s02. What I
want is to allocate the processes always in the same node.
I tried to do this by specifying an MPI_Info object which is then passed to the MPI_Spawn
routine. I tried to set the "host" property to the hostname of the machine
where the scheduler is running but this didn't help.
Unfortunately there is very little documentation on this.
Thanks for the help,
Simone
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
