Hi
Well, with qconf -sq one.q I got the following:
[oscar@aguia free-noise]$ qconf -sq one.q
qname one.q
hostlist compute-1-30.local compute-1-2.local compute-1-3.local
\
compute-1-4.local compute-1-5.local compute-1-6.local \
compute-1-7.local compute-1-8.local compute-1-9.local \
compute-1-10.local compute-1-11.local compute-1-12.local \
compute-1-13.local compute-1-14.local compute-1-15.local
seq_no 0
load_thresholds np_load_avg=1.75
suspend_thresholds NONE
nsuspend 1
suspend_interval 00:05:00
priority 0
min_cpu_interval 00:05:00
processors UNDEFINED
qtype BATCH INTERACTIVE
ckpt_list NONE
pe_list make mpich mpi orte
rerun FALSE
slots 1,[compute-1-30.local=1],[compute-1-2.local=1], \
[compute-1-3.local=1],[compute-1-5.local=1], \
[compute-1-8.local=1],[compute-1-6.local=1], \
[compute-1-4.local=1],[compute-1-9.local=1], \
[compute-1-11.local=1],[compute-1-7.local=1], \
[compute-1-13.local=1],[compute-1-10.local=1], \
[compute-1-15.local=1],[compute-1-12.local=1], \
[compute-1-14.local=1]
the admin was who created this queue, so I have to speak to him to change the
number of slots to number of threads that i wish to use.
Then I could make use of:
===
export OMP_NUM_THREADS=N
mpirun -map-by slot:pe=$OMP_NUM_THREADS -np $(bc <<<"$NSLOTS /
$OMP_NUM_THREADS") ./inverse.exe
===
For now in my case this command line just would work for 10 processes and the
work wouldn't be divided in threads, is it right?
can I set a maximum number of threads in the queue one.q (e.g. 15 ) and change
the number in the 'export' for my convenience
I feel like a child hearing the adults speaking
Thanks I'm learning a lot
Oscar Fabian Mojica Ladino
Geologist M.S. in Geophysics
> From: re...@staff.uni-marburg.de
> Date: Tue, 19 Aug 2014 19:51:46 +0200
> To: us...@open-mpi.org
> Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
>
> Hi,
>
> Am 19.08.2014 um 19:06 schrieb Oscar Mojica:
>
> > I discovered what was the error. I forgot include the '-fopenmp' when I
> > compiled the objects in the Makefile, so the program worked but it didn't
> > divide the job in threads. Now the program is working and I can use until
> > 15 cores for machine in the queue one.q.
> >
> > Anyway i would like to try implement your advice. Well I'm not alone in the
> > cluster so i must implement your second suggestion. The steps are
> >
> > a) Use '$ qconf -mp orte' to change the allocation rule to 8
>
> The number of slots defined in your used one.q was also increased to 8
> (`qconf -sq one.q`)?
>
>
> > b) Set '#$ -pe orte 80' in the script
>
> Fine.
>
>
> > c) I'm not sure how to do this step. I'd appreciate your help here. I can
> > add some lines to the script to determine the PE_HOSTFILE path and
> > contents, but i don't know how alter it
>
> For now you can put in your jobscript (just after OMP_NUM_THREAD is exported):
>
> awk -v omp_num_threads=$OMP_NUM_THREADS '{ $2/=omp_num_threads; print }'
> $PE_HOSTFILE > $TMPDIR/machines
> export PE_HOSTFILE=$TMPDIR/machines
>
> =============
>
> Unfortunately noone stepped into this discussion, as in my opinion it's a
> much broader issue which targets all users who want to combine MPI with
> OpenMP. The queuingsystem should get a proper request for the overall amount
> of slots the user needs. For now this will be forwarded to Open MPI and it
> will use this information to start the appropriate number of processes (which
> was an achievement for the Tight Integration out-of-the-box of course) and
> ignores any setting of OMP_NUM_THREADS. So, where should the generated list
> of machines be adjusted; there are several options:
>
> a) The PE of the queuingsystem should do it:
>
> + a one time setup for the admin
> + in SGE the "start_proc_args" of the PE could alter the $PE_HOSTFILE
> - the "start_proc_args" would need to know the number of threads, i.e.
> OMP_NUM_THREADS must be defined by "qsub -v ..." outside of the jobscript
> (tricky scanning of the submitted jobscript for OMP_NUM_THREADS would be too
> nasty)
> - limits to use inside the jobscript calls to libraries behaving in the same
> way as Open MPI only
>
>
> b) The particular queue should do it in a queue prolog:
>
> same as a) I think
>
>
> c) The user should do it
>
> + no change in the SGE installation
> - each and every user must include it in all the jobscripts to adjust the
> list and export the pointer to the $PE_HOSTFILE, but he could change it forth
> and back for different steps of the jobscript though
>
>
> d) Open MPI should do it
>
> + no change in the SGE installation
> + no change to the jobscript
> + OMP_NUM_THREADS can be altered for different steps of the jobscript while
> staying inside the granted allocation automatically
> o should MKL_NUM_THREADS be covered too (does it use OMP_NUM_THREADS already)?
>
> -- Reuti
>
>
> > echo "PE_HOSTFILE:"
> > echo $PE_HOSTFILE
> > echo
> > echo "cat PE_HOSTFILE:"
> > cat $PE_HOSTFILE
> >
> > Thanks for take a time for answer this emails, your advices had been very
> > useful
> >
> > PS: The version of SGE is OGS/GE 2011.11p1
> >
> >
> > Oscar Fabian Mojica Ladino
> > Geologist M.S. in Geophysics
> >
> >
> > > From: re...@staff.uni-marburg.de
> > > Date: Fri, 15 Aug 2014 20:38:12 +0200
> > > To: us...@open-mpi.org
> > > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
> > >
> > > Hi,
> > >
> > > Am 15.08.2014 um 19:56 schrieb Oscar Mojica:
> > >
> > > > Yes, my installation of Open MPI is SGE-aware. I got the following
> > > >
> > > > [oscar@compute-1-2 ~]$ ompi_info | grep grid
> > > > MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.2)
> > >
> > > Fine.
> > >
> > >
> > > > I'm a bit slow and I didn't understand the las part of your message. So
> > > > i made a test trying to solve my doubts.
> > > > This is the cluster configuration: There are some machines turned off
> > > > but that is no problem
> > > >
> > > > [oscar@aguia free-noise]$ qhost
> > > > HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
> > > > -------------------------------------------------------------------------------
> > > > global - - - - - - -
> > > > compute-1-10 linux-x64 16 0.97 23.6G 558.6M 996.2M 0.0
> > > > compute-1-11 linux-x64 16 - 23.6G - 996.2M -
> > > > compute-1-12 linux-x64 16 0.97 23.6G 561.1M 996.2M 0.0
> > > > compute-1-13 linux-x64 16 0.99 23.6G 558.7M 996.2M 0.0
> > > > compute-1-14 linux-x64 16 1.00 23.6G 555.1M 996.2M 0.0
> > > > compute-1-15 linux-x64 16 0.97 23.6G 555.5M 996.2M 0.0
> > > > compute-1-16 linux-x64 8 0.00 15.7G 296.9M 1000.0M 0.0
> > > > compute-1-17 linux-x64 8 0.00 15.7G 299.4M 1000.0M 0.0
> > > > compute-1-18 linux-x64 8 - 15.7G - 1000.0M -
> > > > compute-1-19 linux-x64 8 - 15.7G - 996.2M -
> > > > compute-1-2 linux-x64 16 1.19 23.6G 468.1M 1000.0M 0.0
> > > > compute-1-20 linux-x64 8 0.04 15.7G 297.2M 1000.0M 0.0
> > > > compute-1-21 linux-x64 8 - 15.7G - 1000.0M -
> > > > compute-1-22 linux-x64 8 0.00 15.7G 297.2M 1000.0M 0.0
> > > > compute-1-23 linux-x64 8 0.16 15.7G 299.6M 1000.0M 0.0
> > > > compute-1-24 linux-x64 8 0.00 15.7G 291.5M 996.2M 0.0
> > > > compute-1-25 linux-x64 8 0.04 15.7G 293.4M 996.2M 0.0
> > > > compute-1-26 linux-x64 8 - 15.7G - 1000.0M -
> > > > compute-1-27 linux-x64 8 0.00 15.7G 297.0M 1000.0M 0.0
> > > > compute-1-29 linux-x64 8 - 15.7G - 1000.0M -
> > > > compute-1-3 linux-x64 16 - 23.6G - 996.2M -
> > > > compute-1-30 linux-x64 16 - 23.6G - 996.2M -
> > > > compute-1-4 linux-x64 16 0.97 23.6G 571.6M 996.2M 0.0
> > > > compute-1-5 linux-x64 16 1.00 23.6G 559.6M 996.2M 0.0
> > > > compute-1-6 linux-x64 16 0.66 23.6G 403.1M 996.2M 0.0
> > > > compute-1-7 linux-x64 16 0.95 23.6G 402.7M 996.2M 0.0
> > > > compute-1-8 linux-x64 16 0.97 23.6G 556.8M 996.2M 0.0
> > > > compute-1-9 linux-x64 16 1.02 23.6G 566.0M 1000.0M 0.0
> > > >
> > > > I ran my program using only MPI with 10 processors of the queue one.q
> > > > which has 14 machines (compute-1-2 to compute-1-15). Whit 'qstat -t' I
> > > > got:
> > > >
> > > > [oscar@aguia free-noise]$ qstat -t
> > > > job-ID prior name user state submit/start at queue master ja-task-ID
> > > > task-ID state cpu mem io stat failed
> > > > -----------------------------------------------------------------------------------------------------------------------------------------------------------------------
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-2.local
> > > > MASTER r 00:49:12 554.13753 0.09163
> > > > one.q@compute-1-2.local SLAVE
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-5.local
> > > > SLAVE 1.compute-1-5 r 00:48:53 551.49022 0.09410
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-9.local
> > > > SLAVE 1.compute-1-9 r 00:50:00 564.22764 0.09409
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-12.local
> > > > SLAVE 1.compute-1-12 r 00:47:30 535.30379 0.09379
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-13.local
> > > > SLAVE 1.compute-1-13 r 00:49:51 561.69868 0.09379
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-14.local
> > > > SLAVE 1.compute-1-14 r 00:49:14 554.60818 0.09379
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-10.local
> > > > SLAVE 1.compute-1-10 r 00:49:59 562.95487 0.09349
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-15.local
> > > > SLAVE 1.compute-1-15 r 00:50:01 563.27221 0.09361
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-8.local
> > > > SLAVE 1.compute-1-8 r 00:49:26 556.68431 0.09349
> > > > 2726 0.50500 job oscar r 08/15/2014 12:38:21 one.q@compute-1-4.local
> > > > SLAVE 1.compute-1-4 r 00:49:27 556.87510 0.04967
> > >
> > > Yes, here you got 10 slots (= cores) granted by SGE. So there is no free
> > > core left inside the allocation of SGE to allow the use of additional
> > > cores for your threads. If you use more cores than granted by SGE, it
> > > will oversubscribe the machines.
> > >
> > > The issue is now:
> > >
> > > a) If you want 8 threads per MPI process, your job will use 80 cores in
> > > total - for now SGE isn't aware of it.
> > >
> > > b) Although you specified $fill_up as allocation rule, it looks like
> > > $round_robin. Is there more than one slot defined in the queue definition
> > > of one.q to get exclusive access?
> > >
> > > c) What version of SGE are you using? Certain ones use cgroups or bind
> > > processes directly to cores (although it usually needs to be requested by
> > > the job: first line of `qconf -help`).
> > >
> > >
> > > In case you are alone in the cluster, you could bypass the allocation
> > > with b) (unless you are hit by c)). But having a mixture of users and
> > > jobs a different handling would be necessary to handle this in a proper
> > > way IMO:
> > >
> > > a) having a PE with a fixed allocation rule of 8
> > >
> > > b) requesting this PE with an overall slot count of 80
> > >
> > > c) copy and alter the $PE_HOSTFILE to show only (granted core count per
> > > machine) divided by (OMP_NUM_THREADS) per entry, change $PE_HOSTFILE so
> > > that it points to the altered file
> > >
> > > d) Open MPI with a Tight Integration will now start only N process per
> > > machine according to the altered hostfile, in your case one
> > >
> > > e) Your application can start the desired threads and you stay inside the
> > > granted allocation
> > >
> > > -- Reuti
> > >
> > >
> > > > I accessed to the MASTER processor with 'ssh compute-1-2.local' , and
> > > > with $ ps -e f and got this, I'm showing only the last lines
> > > >
> > > > 2506 ? Ss 0:00 /usr/sbin/atd
> > > > 2548 tty1 Ss+ 0:00 /sbin/mingetty /dev/tty1
> > > > 2550 tty2 Ss+ 0:00 /sbin/mingetty /dev/tty2
> > > > 2552 tty3 Ss+ 0:00 /sbin/mingetty /dev/tty3
> > > > 2554 tty4 Ss+ 0:00 /sbin/mingetty /dev/tty4
> > > > 2556 tty5 Ss+ 0:00 /sbin/mingetty /dev/tty5
> > > > 2558 tty6 Ss+ 0:00 /sbin/mingetty /dev/tty6
> > > > 3325 ? Sl 0:04 /opt/gridengine/bin/linux-x64/sge_execd
> > > > 17688 ? S 0:00 \_ sge_shepherd-2726 -bg
> > > > 17695 ? Ss 0:00 \_ -bash
> > > > /opt/gridengine/default/spool/compute-1-2/job_scripts/2726
> > > > 17797 ? S 0:00 \_ /usr/bin/time -f %E /opt/openmpi/bin/mpirun -v -np 10
> > > > ./inverse.exe
> > > > 17798 ? S 0:01 \_ /opt/openmpi/bin/mpirun -v -np 10 ./inverse.exe
> > > > 17799 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-5.local PATH=/opt/openmpi/bin:$PATH ; expo
> > > > 17800 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-9.local PATH=/opt/openmpi/bin:$PATH ; expo
> > > > 17801 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-12.local PATH=/opt/openmpi/bin:$PATH ; exp
> > > > 17802 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-13.local PATH=/opt/openmpi/bin:$PATH ; exp
> > > > 17803 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-14.local PATH=/opt/openmpi/bin:$PATH ; exp
> > > > 17804 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-10.local PATH=/opt/openmpi/bin:$PATH ; exp
> > > > 17805 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-15.local PATH=/opt/openmpi/bin:$PATH ; exp
> > > > 17806 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-8.local PATH=/opt/openmpi/bin:$PATH ; expo
> > > > 17807 ? Sl 0:00 \_ /opt/gridengine/bin/linux-x64/qrsh -inherit -nostdin
> > > > -V compute-1-4.local PATH=/opt/openmpi/bin:$PATH ; expo
> > > > 17826 ? R 31:36 \_ ./inverse.exe
> > > > 3429 ? Ssl 0:00 automount --pid-file /var/run/autofs.pid
> > > >
> > > > So the job is using the 10 machines, Until here is all right OK. Do you
> > > > think that changing the "allocation_rule " to a number instead $fill_up
> > > > the MPI processes would divide the work in that number of threads?
> > > >
> > > > Thanks a lot
> > > >
> > > > Oscar Fabian Mojica Ladino
> > > > Geologist M.S. in Geophysics
> > > >
> > > >
> > > > PS: I have another doubt, what is a slot? is a physical core?
> > > >
> > > >
> > > > > From: re...@staff.uni-marburg.de
> > > > > Date: Thu, 14 Aug 2014 23:54:22 +0200
> > > > > To: us...@open-mpi.org
> > > > > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
> > > > >
> > > > > Hi,
> > > > >
> > > > > I think this is a broader issue in case an MPI library is used in
> > > > > conjunction with threads while running inside a queuing system.
> > > > > First: whether your actual installation of Open MPI is SGE-aware you
> > > > > can check with:
> > > > >
> > > > > $ ompi_info | grep grid
> > > > > MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.6.5)
> > > > >
> > > > > Then we can look at the definition of your PE: "allocation_rule
> > > > > $fill_up". This means that SGE will grant you 14 slots in total in
> > > > > any combination on the available machines, means 8+4+2 slots
> > > > > allocation is an allowed combination like 4+4+3+3 and so on.
> > > > > Depending on the SGE-awareness it's a question: will your application
> > > > > just start processes on all nodes and completely disregard the
> > > > > granted allocation, or as the other extreme does it stays on one and
> > > > > the same machine for all started processes? On the master node of the
> > > > > parallel job you can issue:
> > > > >
> > > > > $ ps -e f
> > > > >
> > > > > (f w/o -) to have a look whether `ssh` or `qrsh -inhert ...` is used
> > > > > to reach other machines and their requested process count.
> > > > >
> > > > >
> > > > > Now to the common problem in such a set up:
> > > > >
> > > > > AFAICS: for now there is no way in the Open MPI + SGE combination to
> > > > > specify the number of MPI processes and intended number of threads
> > > > > which are automatically read by Open MPI while staying inside the
> > > > > granted slot count and allocation. So it seems to be necessary to
> > > > > have the intended number of threads being honored by Open MPI too.
> > > > >
> > > > > Hence specifying e.g. "allocation_rule 8" in such a setup while
> > > > > requesting 32 processes, would for now start 32 processes by MPI
> > > > > already, as Open MP reads the $PE_HOSTFILE and acts accordingly.
> > > > >
> > > > > Open MPI would have to read the generated machine file in a slightly
> > > > > different way regarding threads: a) read the $PE_HOSTFILE, b) divide
> > > > > the granted slots per machine by OMP_NUM_THREADS, c) throw an error
> > > > > in case it's not divisible by OMP_NUM_THREADS. Then start one process
> > > > > per quotient.
> > > > >
> > > > > Would this work for you?
> > > > >
> > > > > -- Reuti
> > > > >
> > > > > PS: This would also mean to have a couple of PEs in SGE having a
> > > > > fixed "allocation_rule". While this works right now, an extension in
> > > > > SGE could be "$fill_up_omp"/"$round_robin_omp" and using
> > > > > OMP_NUM_THREADS there too, hence it must not be specified as an
> > > > > `export` in the job script but either on the command line or inside
> > > > > the job script in #$ lines as job requests. This would mean to
> > > > > collect slots in bunches of OMP_NUM_THREADS on each machine to reach
> > > > > the overall specified slot count. Whether OMP_NUM_THREADS or n times
> > > > > OMP_NUM_THREADS is allowed per machine needs to be discussed.
> > > > >
> > > > > PS2: As Univa SGE can also supply a list of granted cores in the
> > > > > $PE_HOSTFILE, it would be an extension to feed this to Open MPI to
> > > > > allow any UGE aware binding.
> > > > >
> > > > >
> > > > > Am 14.08.2014 um 21:52 schrieb Oscar Mojica:
> > > > >
> > > > > > Guys
> > > > > >
> > > > > > I changed the line to run the program in the script with both
> > > > > > options
> > > > > > /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-none -np
> > > > > > $NSLOTS ./inverse.exe
> > > > > > /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v --bind-to-socket
> > > > > > -np $NSLOTS ./inverse.exe
> > > > > >
> > > > > > but I got the same results. When I use man mpirun appears:
> > > > > >
> > > > > > -bind-to-none, --bind-to-none
> > > > > > Do not bind processes. (Default.)
> > > > > >
> > > > > > and the output of 'qconf -sp orte' is
> > > > > >
> > > > > > pe_name orte
> > > > > > slots 9999
> > > > > > user_lists NONE
> > > > > > xuser_lists NONE
> > > > > > start_proc_args /bin/true
> > > > > > stop_proc_args /bin/true
> > > > > > allocation_rule $fill_up
> > > > > > control_slaves TRUE
> > > > > > job_is_first_task FALSE
> > > > > > urgency_slots min
> > > > > > accounting_summary TRUE
> > > > > >
> > > > > > I don't know if the installed Open MPI was compiled with
> > > > > > '--with-sge'. How can i know that?
> > > > > > before to think in an hybrid application i was using only MPI and
> > > > > > the program used few processors (14). The cluster possesses 28
> > > > > > machines, 15 with 16 cores and 13 with 8 cores totalizing 344 units
> > > > > > of processing. When I submitted the job (only MPI), the MPI
> > > > > > processes were spread to the cores directly, for that reason I
> > > > > > created a new queue with 14 machines trying to gain more time. the
> > > > > > results were the same in both cases. In the last case i could prove
> > > > > > that the processes were distributed to all machines correctly.
> > > > > >
> > > > > > What I must to do?
> > > > > > Thanks
> > > > > >
> > > > > > Oscar Fabian Mojica Ladino
> > > > > > Geologist M.S. in Geophysics
> > > > > >
> > > > > >
> > > > > > > Date: Thu, 14 Aug 2014 10:10:17 -0400
> > > > > > > From: maxime.boissonnea...@calculquebec.ca
> > > > > > > To: us...@open-mpi.org
> > > > > > > Subject: Re: [OMPI users] Running a hybrid MPI+openMP program
> > > > > > >
> > > > > > > Hi,
> > > > > > > You DEFINITELY need to disable OpenMPI's new default binding.
> > > > > > > Otherwise,
> > > > > > > your N threads will run on a single core. --bind-to socket would
> > > > > > > be my
> > > > > > > recommendation for hybrid jobs.
> > > > > > >
> > > > > > > Maxime
> > > > > > >
> > > > > > >
> > > > > > > Le 2014-08-14 10:04, Jeff Squyres (jsquyres) a écrit :
> > > > > > > > I don't know much about OpenMP, but do you need to disable Open
> > > > > > > > MPI's default bind-to-core functionality (I'm assuming you're
> > > > > > > > using Open MPI 1.8.x)?
> > > > > > > >
> > > > > > > > You can try "mpirun --bind-to none ...", which will have Open
> > > > > > > > MPI not bind MPI processes to cores, which might allow OpenMP
> > > > > > > > to think that it can use all the cores, and therefore it will
> > > > > > > > spawn num_cores threads...?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Aug 14, 2014, at 9:50 AM, Oscar Mojica
> > > > > > > > <o_moji...@hotmail.com> wrote:
> > > > > > > >
> > > > > > > >> Hello everybody
> > > > > > > >>
> > > > > > > >> I am trying to run a hybrid mpi + openmp program in a cluster.
> > > > > > > >> I created a queue with 14 machines, each one with 16 cores.
> > > > > > > >> The program divides the work among the 14 processors with MPI
> > > > > > > >> and within each processor a loop is also divided into 8
> > > > > > > >> threads for example, using openmp. The problem is that when I
> > > > > > > >> submit the job to the queue the MPI processes don't divide the
> > > > > > > >> work into threads and the program prints the number of threads
> > > > > > > >> that are working within each process as one.
> > > > > > > >>
> > > > > > > >> I made a simple test program that uses openmp and I logged in
> > > > > > > >> one machine of the fourteen. I compiled it using gfortran
> > > > > > > >> -fopenmp program.f -o exe, set the OMP_NUM_THREADS environment
> > > > > > > >> variable equal to 8 and when I ran directly in the terminal
> > > > > > > >> the loop was effectively divided among the cores and for
> > > > > > > >> example in this case the program printed the number of threads
> > > > > > > >> equal to 8
> > > > > > > >>
> > > > > > > >> This is my Makefile
> > > > > > > >>
> > > > > > > >> # Start of the makefile
> > > > > > > >> # Defining variables
> > > > > > > >> objects = inv_grav3d.o funcpdf.o gr3dprm.o fdjac.o dsvd.o
> > > > > > > >> #f90comp = /opt/openmpi/bin/mpif90
> > > > > > > >> f90comp = /usr/bin/mpif90
> > > > > > > >> #switch = -O3
> > > > > > > >> executable = inverse.exe
> > > > > > > >> # Makefile
> > > > > > > >> all : $(executable)
> > > > > > > >> $(executable) : $(objects)
> > > > > > > >> $(f90comp) -fopenmp -g -O -o $(executable) $(objects)
> > > > > > > >> rm $(objects)
> > > > > > > >> %.o: %.f
> > > > > > > >> $(f90comp) -c $<
> > > > > > > >> # Cleaning everything
> > > > > > > >> clean:
> > > > > > > >> rm $(executable)
> > > > > > > >> # rm $(objects)
> > > > > > > >> # End of the makefile
> > > > > > > >>
> > > > > > > >> and the script that i am using is
> > > > > > > >>
> > > > > > > >> #!/bin/bash
> > > > > > > >> #$ -cwd
> > > > > > > >> #$ -j y
> > > > > > > >> #$ -S /bin/bash
> > > > > > > >> #$ -pe orte 14
> > > > > > > >> #$ -N job
> > > > > > > >> #$ -q new.q
> > > > > > > >>
> > > > > > > >> export OMP_NUM_THREADS=8
> > > > > > > >> /usr/bin/time -f "%E" /opt/openmpi/bin/mpirun -v -np $NSLOTS
> > > > > > > >> ./inverse.exe
> > > > > > > >>
> > > > > > > >> am I forgetting something?
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >>
> > > > > > > >> Oscar Fabian Mojica Ladino
> > > > > > > >> Geologist M.S. in Geophysics
> > > > > > > >> _______________________________________________
> > > > > > > >> users mailing list
> > > > > > > >> us...@open-mpi.org
> > > > > > > >> Subscription:
> > > > > > > >> http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > > > > >> Link to this post:
> > > > > > > >> http://www.open-mpi.org/community/lists/users/2014/08/25016.php
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > ---------------------------------
> > > > > > > Maxime Boissonneault
> > > > > > > Analyste de calcul - Calcul Québec, Université Laval
> > > > > > > Ph. D. en physique
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > users mailing list
> > > > > > > us...@open-mpi.org
> > > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > > > > Link to this post:
> > > > > > > http://www.open-mpi.org/community/lists/users/2014/08/25020.php
> > > > > > _______________________________________________
> > > > > > users mailing list
> > > > > > us...@open-mpi.org
> > > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > > > Link to this post:
> > > > > > http://www.open-mpi.org/community/lists/users/2014/08/25032.php
> > > > >
> > > > > _______________________________________________
> > > > > users mailing list
> > > > > us...@open-mpi.org
> > > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > > Link to this post:
> > > > > http://www.open-mpi.org/community/lists/users/2014/08/25034.php
> > > > _______________________________________________
> > > > users mailing list
> > > > us...@open-mpi.org
> > > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > > Link to this post:
> > > > http://www.open-mpi.org/community/lists/users/2014/08/25037.php
> > >
> > > _______________________________________________
> > > users mailing list
> > > us...@open-mpi.org
> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > > Link to this post:
> > > http://www.open-mpi.org/community/lists/users/2014/08/25038.php
> > _______________________________________________
> > users mailing list
> > us...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> > Link to this post:
> > http://www.open-mpi.org/community/lists/users/2014/08/25079.php
>
> _______________________________________________
> users mailing list
> us...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
> Link to this post:
> http://www.open-mpi.org/community/lists/users/2014/08/25080.php
