Hello,
> The other issue seems to be, that in fact your job is using only one
machine, which means that it is essentially ignoring any granted slot
allocation. While the job is running, can you please execute on the
master node of the parallel job:
>
> $ ps -e f
>
> (f w/o -) and post the relevant lines belonging to either sge_execd or
just running as kids of the init process, in case they jumped out of the
process tree. Maybe a good start would be to execute something like
`mpiexec sleep 300` in the jobscript.
>
i invoked
qsub -pe orte 160 -j yes -cwd -S /bin/bash <<< "mpiexec -n 160 sleep 300"
the only line ('ps -e f') on the master node was:
55722 ? Sl 3:42 /opt/sge/bin/lx-amd64/sge_qmaster
No other sge lines, no child processes from it, and no other init
processes leading to sge While at the same time the sleep processes were
running on the nodes (Checked with ps command on the nodes).
The qstat command gave :
264 0.60500 STDIN ulrich r 08/15/2016 11:33:02
all.q@exec-node01 MASTER
all.q@exec-node01 SLAVE
all.q@exec-node01 SLAVE
all.q@exec-node01 SLAVE
[ ...]
264 0.60500 STDIN ulrich r 08/15/2016 11:33:02
all.q@exec-node03 SLAVE
all.q@exec-node03 SLAVE
all.q@exec-node03 SLAVE
[ ... ]
264 0.60500 STDIN ulrich r 08/15/2016 11:33:02
all.q@exec-node05 SLAVE
all.q@exec-node05 SLAVE
[ ...]
Because there was only the master deamon running on the master node, and
you were tlaking about child processes: Was this now normal behaviour my
cluster showed or is there something wrong?
Kind reagrds, ulrich
On 08/12/2016 07:11 PM, Reuti wrote:
> Hi,
>
>> Am 12.08.2016 um 18:48 schrieb Ulrich Hiller <hil...@mpia-hd.mpg.de>:
>>
>> Hello,
>>
>> i have a strange effect, where i am not sure whether it is "only" a
>> misconfiguration or a bug.
>>
>> First: I run son of gridengine 8.1.9-1.el6.x86_64 (i installed the rhel
>> rpm on an opensuse 13.1 machine. This should not matter in this case,
>> and it is reported to be able to run on opensuse).
>>
>> mpirun and mpiexec are from openmpi-1.10.3 (no other mpi was installed,
>> neither on master, nor on slaves). The installation was made with:
>> ./configure --prefix=`pwd`/build --disable-dlopen --disable-mca-dso
>> --with-orte --with-sge --with-x --enable-mpi-thread-multiple
>> --enable-orterun-prefix-by-default --enable-mpirun-prefix-by-default
>> --enable-orte-static-ports --enable-mpi-cxx --enable-mpi-cxx-seek
>> --enable-oshmem --enable-java --enable-mpi-java
>> make
>> make install
>>
>> I attached the outputs of 'qconf -ap all.q' , 'qconf -sconf' and 'qconf
>> -sp orte' as textfiles.
>>
>> Now my problem:
>> I asked for 20 cores and if i run qstat -u '*' it shows that this job
>> is being run in slave07 using 20 cores but is not true! if i run qstat
>> -f -u '*' i see that this job is only using 3 cores in salve07 and
>> there are 17 cores in other nodes allocated to this job which are in fact
>> unused!
>
> qstat will list only the master node of the parallel job and the number of
> overall slots. The granted allocation you can check with:
>
> $ qstat -g t -u '*'
>
> The other issue seems to be, that in fact your job is using only one machine,
> which means that it is essentially ignoring any granted slot allocation.
> While the job is running, can you please execute on the master node of the
> parallel job:
>
> $ ps -e f
>
> (f w/o -) and post the relevant lines belonging to either sge_execd or just
> running as kids of the init process, in case they jumped out of the process
> tree. Maybe a good start would be to execute something like `mpiexec sleep
> 300` in the jobscript.
>
> Next step could be a `mpihello.c` where you put an almost endless loop inside
> and switch off all optimizations during compilations to check whether these
> slave processes are distributed in the correct way.
>
> Note that some applications will check the number of cores they are running
> on and start by OpenMP (not Open MPI) as many threads as cores are found.
> Could this be the case for your application too?
>
> -- Reuti
>
>
>> Or other example:
>> My job took say 6 cpus on slave07 and 14 on slave06 but nothing was
>> running on 06 and therefore a waste of ressource on 06 and overload on
>> 07 becomes highly possible (the numbers are made up).
>> If i ran 1 Cpus in many independent jobs that would not be an issue, but
>> imagine i now request 60 cpus on slave07, that would seriously overload
>> the node in many cases.
>>
>> Or other example:
>> if i ask for say 50 CPUs, the job will start on one node, e.g,
>> slave01, but only reserving say 15 CPUs out of 64 and reserve the rest
>> on many other nodes (obviously wasting space doing nothing).
>> This has the bad consequence of allocating many more CPUs than available
>> when many jobs are running, imagine you have 10 jobs like this one...
>> some nodes will run maybe 3 even if they only have 24 CPUs...
>>
>> I hope that i have made clear what the issue is.
>>
>> I also see that the `qstat` and `qstat -f` are in disagreement. The
>> latter is correct, i checked the processes running on the nodes.
>>
>>
>> Did somebody already encounter such a problem? Does somebody have an
>> idea where to look into or what to test?
>>
>> With kind regards, ulrich
>>
>>
>>
>> <qhost.txt><qconf-sconf.txt><qconf-mp-orte.txt><qconf-all.q>_______________________________________________
>> users mailing list
>> users@gridengine.org
>> https://gridengine.org/mailman/listinfo/users
>
_______________________________________________
users mailing list
users@gridengine.org
https://gridengine.org/mailman/listinfo/users
