OK. The two different partitions I saw was due to not specifying partition
name for the first set (before packjob). Here is a better script
#!/bin/bash
#SBATCH --job-name=myQE
#SBATCH --output=big-mem
#SBATCH --mem-per-cpu=16g --ntasks=2
#SBATCH -N 1
#SBATCH --partition=QUARTZ
#SBATCH --account=z5
#SBATCH packjob
#SBATCH --mem-per-cpu=10g --ntasks=4
#SBATCH -N 1
#SBATCH --partition=QUARTZ
#SBATCH --account=z5
srun pw.x -i mos2.rlx.in
One node should run 2 processes (32GB total) and one other node should run
4 process (40GB total).
The queue looks like
$ squeue
JOBID PARTITION NAME USER ST TIME NODES
NODELIST(REASON)
747+1 QUARTZ myQE ghatee R 0:02 1 rocks7
747+0 QUARTZ myQE ghatee R 0:02 1
compute-0-2
As I checked the node, one 2 processes are running on compute-0-2 (first
set before packjob). But there is no processes on rocks7.
$ rocks run host compute-0-2 "ps aux | grep pw.x"
ghatee 30234 0.0 0.0 251208 4996 ? Sl 15:04 0:00 srun pw.x
-i mos2.rlx.in
ghatee 30235 0.0 0.0 46452 748 ? S 15:04 0:00 srun pw.x
-i mos2.rlx.in
ghatee 30247 99.8 0.1 1930484 129696 ? Rl 15:04 4:31
/home/ghatee/QuantumEspresso621/bin/pw.x -i mos2.rlx.in
ghatee 30248 99.8 0.1 1930488 129704 ? Rl 15:04 4:31
/home/ghatee/QuantumEspresso621/bin/pw.x -i mos2.rlx.in
ghatee 30352 0.0 0.0 113132 1592 ? Ss 15:09 0:00 bash -c ps
aux | grep pw.x
ghatee 30381 0.0 0.0 112664 960 ? S 15:09 0:00 grep pw.x
$ rocks run host rocks7 "ps aux | grep pw.x"
ghatee 17141 0.0 0.0 316476 26632 pts/21 Sl+ 23:39 0:00
/opt/rocks/bin/python /opt/rocks/bin/rocks run host rocks7 ps aux | grep
pw.x
ghatee 17143 0.0 0.0 113132 1364 pts/21 S+ 23:39 0:00 bash -c ps
aux | grep pw.x
ghatee 17145 0.0 0.0 112664 960 pts/21 R+ 23:39 0:00 grep pw.x
Any idea?
It seems that the mpirun I have is not compatible with the hetro
configuration because the SBATCH parameters are straight forward.
Regards,
Mahmood
On Wed, Mar 27, 2019 at 11:03 PM Christopher Samuel <ch...@csamuel.org>
wrote:
> On 3/27/19 11:29 AM, Mahmood Naderan wrote:
>
> > Thank you very much. you are right. I got it.
>
> Cool, good to hear.
>
> I'd love to hear whether you get heterogenous MPI jobs working too!
>
> All the best,
> Chris
> --
> Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
>
>
