Dear Ralph,
I am copying your email from the web site because I had enabled the
option to receive all the emails once per day
On 11/04/2012 05:27 PM, George Markomanolis wrote:
> Dear all,
>
> I am trying to execute an experiment by oversubscribing the nodes. So I
have available some clusters (I can use up to 8-10 different clusters
during one execution) and I have totally around to 1300 cores. I am
executing the EP benchmark from the NAS suite which means that there
are not a lot of MPI messages, just some collective MPI calls.
>
> The number of the MPI processes per node, depends on the available
memory of each node. Thus in the machinefile I have declared one node
13 times if I want 13 MPI processes on it. Is that correct?
You *can* do it that way, or you could just use "slots=13" for that
node in the file, and list it only once.
OK, but I assume the result is the same, right?
> Giving a machinefile of 32768 nodes when I want to execute 32768 processes, does OpenMPI
behave like there is no oversubscribing?
Yes, it should - I assume you mean "slots" and not "nodes" in the
above statement, since you indicate that you listed each node multiple
times to set the number of slots on that node.
Yes, I mean slots.
> If yes how can I give a machinefile where there is different number of MPI processes on each
node? The maximum number of MPI processes that I have in a node is 388.
Just assign the number of slots on each node to be the number of
processes you want on that node
OK
>
> My problem is that I can execute 16384 processes but not 32768. In
the first case I need around to 3 minutes for the execution but in the
second case, even after 7 hours the benchmark does not even start.
There is no error, I am just cancelling the job by myself but I am
assuming that something is wrong because 7 hours it is too much. I
have to say that I executed the instance of 16384 processes without
any problem. I added some debug info in the benchmark and I can see
that the execution is delayed during MPI_Init, it never passes this
point. For the instance of 16384 processes I need around to 2 minutes
to finish the MPI_Init call. I am checking the memory of all the nodes
and there is at least 0.5GB free memory on each node.
>
> I know about the parameter mpi_yield_when_idle but I have read that if
there are not a lot of MPI messages will not improve the performance.
I tried though and nothing changed. I tried also the
mpi_preconnect_mpi just in case but again nothing. Could you please
indicate a reason why is this happening?
You indicated that these jobs are actually spanning multiple clusters
- true? If so, when you cross that 16384 boundary, do you also cross
clusters? Is it possible one or more of the additional clusters is
blocking communications?
I have tried both configurations even using exactly the same nodes with
less MPI processes per node in order to check if a site is blocking the
rest ones and I have tried the half machinefile for the instance of
16384 in order to see if there is any issue by using so many MPI
processes per node. Both were executed fine with the instance of 16384
MPI processes. Also I tried to combine different quarters of the
machinefile in order to check if there is any issue with the combination
of specific sites and again I didn't have a problem.
>
> Moreover I used just one node with 48GB memory in order to execute
2048 MPI processes without any problem, of course I just had to wait a
lot.
>
> I am using OpenMPI v1.4.1 and all the clusters are 64 bit.
>
> I execute the benchmark with the following command:
> mpirun --mca pml ob1 --mca btl tcp,self --mca btl_tcp_if_exclude
ib0,lo,myri0 -machinefile machines -np 32768 ep.D.32768
You could just leave off the "-np N" part of the command line - we'll
assign one process to every slot specified in the machinefile.
OK, nice
Best regards,
George Markomanolis
>
> Best regards,
> George Markomanolis
> _______________________________________________
> users mailing list
> users_at_[hidden]
>http://www.open-mpi.org/mailman/listinfo.cgi/users
